Meta Llama 3.1 8B Instruct FP16

🚀 Meta Llama 3.1

Meta Llama 3.1 is a collection of multilingual large language models, offering various sizes and optimized for multilingual dialogue use cases, outperforming many existing models on common benchmarks.

🚀 Quick Start

This repository contains two versions of Meta-Llama-3.1-8B-Instruct, for use with transformers and with the original llama codebase.

Use with transformers

Starting with transformers >= 4.43.0 onward, you can run conversational inference using the Transformers pipeline abstraction or by leveraging the Auto classes with the generate() function.

Make sure to update your transformers installation via pip install --upgrade transformers.

import transformers
import torch

model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

outputs = pipeline(
    messages,
    max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])

Note: You can also find detailed recipes on how to use the model locally, with torch.compile(), assisted generations, quantised and more at huggingface-llama-recipes

Tool use with transformers

LLaMA-3.1 supports multiple tool use formats. You can see a full guide to prompt formatting here.

Tool use is also supported through chat templates in Transformers. Here is a quick example showing a single simple tool:

# First, define a tool
def get_current_temperature(location: str) -> float:
    """
    Get the current temperature at a location.
    
    Args:
        location: The location to get the temperature for, in the format "City, Country"
    Returns:
        The current temperature at the specified location.
    """
    # This is a mock function. In a real - world scenario, you'd call an API.
    return 25.0

✨ Features

• Multilingual Support: Supports English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.
• Optimized for Dialogue: Instruction tuned text - only models are optimized for multilingual dialogue use cases.
• High Performance: Outperforms many of the available open - source and closed chat models on common industry benchmarks.
• Multiple Sizes: Available in 8B, 70B, and 405B sizes.
• Improved Inference Scalability: All model versions use Grouped - Query Attention (GQA).

📦 Installation

Make sure to update your transformers installation via pip install --upgrade transformers.

💻 Usage Examples

Basic Usage

import transformers
import torch

model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

outputs = pipeline(
    messages,
    max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])

Advanced Usage

# First, define a tool
def get_current_temperature(location: str) -> float:
    """
    Get the current temperature at a location.
    
    Args:
        location: The location to get the temperature for, in the format "City, Country"
    Returns:
        The current temperature at the specified location.
    """
    # This is a mock function. In a real - world scenario, you'd call an API.
    return 25.0

📚 Documentation

Model Information

The Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models in 8B, 70B and 405B sizes (text in/text out). The Llama 3.1 instruction tuned text only models (8B, 70B, 405B) are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks.

Model developer: Meta

Model Architecture: Llama 3.1 is an auto - regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine - tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety.

Supported languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.

Llama 3.1 family of models: Token counts refer to pretraining data only. All model versions use Grouped - Query Attention (GQA) for improved inference scalability.

Model Release Date: July 23, 2024.

Status: This is a static model trained on an offline dataset. Future versions of the tuned models will be released as we improve model safety with community feedback.

License: A custom commercial license, the Llama 3.1 Community License, is available at: [https://github.com/meta - llama/llama - models/blob/main/models/llama3_1/LICENSE](https://github.com/meta - llama/llama - models/blob/main/models/llama3_1/LICENSE)

Where to send questions or comments about the model: Instructions on how to provide feedback or comments on the model can be found in the model [README](https://github.com/meta - llama/llama3). For more technical information about generation parameters and recipes for how to use Llama 3.1 in applications, please go [here](https://github.com/meta - llama/llama - recipes).

Intended Use

Intended Use Cases: Llama 3.1 is intended for commercial and research use in multiple languages. Instruction tuned text only models are intended for assistant - like chat, whereas pretrained models can be adapted for a variety of natural language generation tasks. The Llama 3.1 model collection also supports the ability to leverage the outputs of its models to improve other models including synthetic data generation and distillation. The Llama 3.1 Community License allows for these use cases.

Out - of - scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in any other way that is prohibited by the Acceptable Use Policy and Llama 3.1 Community License. Use in languages beyond those explicitly referenced as supported in this model card.

⚠️ Important Note

Llama 3.1 has been trained on a broader collection of languages than the 8 supported languages. Developers may fine - tune Llama 3.1 models for languages beyond the 8 supported languages provided they comply with the Llama 3.1 Community License and the Acceptable Use Policy and in such cases are responsible for ensuring that any uses of Llama 3.1 in additional languages is done in a safe and responsible manner.

Llama 3.1 Community License Agreement

The full license agreement can be found in the provided text. It includes details about license rights, redistribution, additional commercial terms, disclaimer of warranty, limitation of liability, intellectual property, term and termination, governing law, and jurisdiction.

Llama 3.1 Acceptable Use Policy

Meta is committed to promoting safe and fair use of its tools and features, including Llama 3.1. The policy details prohibited uses such as violating laws, engaging in harmful activities, deceiving others, etc. You can find the most recent copy of this policy at [https://llama.meta.com/llama3_1/use - policy](https://llama.meta.com/llama3_1/use - policy)

📄 License

A custom commercial license, the Llama 3.1 Community License, is available at: [https://github.com/meta - llama/llama - models/blob/main/models/llama3_1/LICENSE](https://github.com/meta - llama/llama - models/blob/main/models/llama3_1/LICENSE)