Llama 4 Scout 17B 16E Instruct FP8

🚀 Llama 4 - Multimodal AI Models

Llama 4 is a collection of natively multimodal AI models developed by Meta. These models offer industry - leading performance in text and image understanding, enabling text and multimodal experiences. They leverage a mixture - of - experts architecture and early fusion for native multimodality.

🚀 Quick Start

Prerequisites

Please make sure you have transformers v4.51.0 installed, or upgrade using pip install -U transformers.

Example Code

from transformers import AutoProcessor, Llama4ForConditionalGeneration
import torch

model_id = "meta-llama/Llama-4-Scout-17B-16E-Instruct"

processor = AutoProcessor.from_pretrained(model_id)
model = Llama4ForConditionalGeneration.from_pretrained(
    model_id,
    attn_implementation="flex_attention",
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

url1 = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
url2 = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/datasets/cat_style_layout.png"
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": url1},
            {"type": "image", "url": url2},
            {"type": "text", "text": "Can you describe how these two images are similar, and how they differ?"},
        ]
    },
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
)

response = processor.batch_decode(outputs[:, inputs["input_ids"].shape[-1]:])[0]
print(response)
print(outputs[0])

✨ Features

• Multimodal Capabilities: Llama 4 models support both text and image understanding, enabling a wide range of multimodal applications such as visual reasoning, captioning, and answering questions about images.
• Mixture - of - Experts Architecture: Leveraging this architecture, the models offer industry - leading performance in various tasks, including reasoning, knowledge, code generation, and multilingual processing.
• Two Efficient Models: The Llama 4 series includes Llama 4 Scout (17 billion parameters, 16 experts) and Llama 4 Maverick (17 billion parameters, 128 experts).

📦 Installation

To use Llama 4 with the transformers library, ensure you have transformers v4.51.0 installed. You can upgrade it using the following command:

pip install -U transformers

💻 Usage Examples

Basic Usage

The provided Python code in the "Quick Start" section demonstrates a basic usage example of using Llama 4 for multimodal input processing and generation.

📚 Documentation

Model Information

• Model Developer: Meta
• Model Architecture: Auto - regressive language models using a mixture - of - experts (MoE) architecture and early fusion for native multimodality.

Intended Use

• Intended Use Cases: Commercial and research use in multiple languages. Instruction - tuned models are for assistant - like chat and visual reasoning tasks. Pretrained models can be adapted for natural language generation. The models also support leveraging outputs for improving other models, including synthetic data generation and distillation.
• Out - of - scope: Any use that violates applicable laws or regulations, the Acceptable Use Policy, or the Llama 4 Community License. Use in languages or capabilities beyond the explicitly supported ones.

Hardware and Software

• Training Factors: Custom training libraries, Meta's custom - built GPU clusters, and production infrastructure were used for pretraining. Fine - tuning, quantization, annotation, and evaluation were also performed on production infrastructure.
• Training Energy Use: Model pre - training utilized a cumulative of 7.38M GPU hours of computation on H100 - 80GB (TDP of 700W) type hardware.

The methodology for determining training energy use and greenhouse gas emissions can be found here.

Benchmarks

Pre - trained models

Instruction tuned models

^reported numbers for MMMU Pro is the average of Standard and Vision tasks

Quantization

• The Llama 4 Scout model is released as BF16 weights and can fit within a single H100 GPU with on - the - fly int4 quantization.
• The Llama 4 Maverick model is released as both BF16 and FP8 quantized weights. The FP8 quantized weights fit on a single H100 DGX host while maintaining quality. Code for on - the - fly int4 quantization is also provided to minimize performance degradation.

Safeguar

The Llama 4 models are subject to the Llama 4 Community License Agreement. When using or distributing the models, please ensure compliance with the license terms, including providing a copy of the agreement, displaying the "Built with Llama" notice, and adhering to the Acceptable Use Policy.

Important Notes

⚠️ Important Note

• Llama 4 has been trained on a broader collection of languages than the 12 supported languages. Developers may fine - tune the models for additional languages, but they must comply with the Llama 4 Community License and the Acceptable Use Policy.
• Llama 4 has been tested for image understanding up to 5 input images. If using additional image understanding capabilities, developers are responsible for risk mitigation, additional testing, and tuning tailored to their specific applications.

💡 Usage Tip

When using Llama 4, make sure to follow the instructions in the Llama [README](https://github.com/meta - llama/llama - models/blob/main/README.md) for providing feedback and the [Llama cookbook](https://github.com/meta - llama/llama - cookbook) for more technical information about generation parameters and usage recipes.