Llama 3.1 Minitron 4B Width Base

🚀 Llama-3.1-Minitron-4B-Width-Base

Llama-3.1-Minitron-4B-Width-Base is a base text - to - text model. It can be applied to various natural language generation tasks, offering a practical solution for commercial use.

✨ Features

• Derived from Llama - 3.1: Obtained by pruning Llama - 3.1 - 8B, specifically pruning the model embedding size and MLP intermediate dimension.
• Continued Training with Distillation: After pruning, continued training with distillation is performed using 94 billion tokens from the continuous pre - training data corpus of Nemotron - 4 15B.
• Model Architecture: Utilizes a model embedding size of 3072, 32 attention heads, an MLP intermediate dimension of 9216, and 32 layers in total. It also employs Grouped - Query Attention (GQA) and Rotary Position Embeddings (RoPE).
• Commercial - Ready: The model is ready for commercial use.

📦 Installation

Support for this model will be added in the upcoming transformers release. In the meantime, you can install the library from source:

pip install git+https://github.com/huggingface/transformers

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoTokenizer, LlamaForCausalLM

# Load the tokenizer and model
model_path = "nvidia/Llama-3.1-Minitron-4B-Width-Base"
tokenizer = AutoTokenizer.from_pretrained(model_path)

device = 'cuda'
dtype = torch.bfloat16
model = LlamaForCausalLM.from_pretrained(model_path, torch_dtype=dtype, device_map=device)

# Prepare the input text
prompt = 'Complete the paragraph: our solar system is'
inputs = tokenizer.encode(prompt, return_tensors='pt').to(model.device)

# Generate the output
outputs = model.generate(inputs, max_length=20)

# Decode and print the output
output_text = tokenizer.decode(outputs[0])
print(output_text)

📚 Documentation

Model Overview

Llama-3.1-Minitron-4B-Width-Base is a base text - to - text model suitable for various natural language generation tasks. It is derived from pruning Llama - 3.1 - 8B, followed by continued training with distillation using 94 billion tokens. This model is ready for commercial use.

Model Architecture

Software Integration

Runtime Engine(s):

• NeMo 24.05

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
• NVIDIA Blackwell
• NVIDIA Hopper
• NVIDIA Lovelace

[Preferred/Supported] Operating System(s):

• Linux

Dataset & Training

Evaluation Results

Overview

5 - shot performance. Language Understanding evaluated using Massive Multitask Language Understanding:

Zero - shot performance. Evaluated using select datasets from the [LM Evaluation Harness](https://github.com/EleutherAI/lm - evaluation - harness) with additions:

Code generation performance. Evaluated using [MBPP](https://github.com/google - research/google - research/tree/master/mbpp):

Inference

Limitations

The model was trained on data containing toxic language, unsafe content, and societal biases originally crawled from the internet. Thus, it may amplify those biases and return toxic responses, especially when prompted with toxic prompts. It may also generate inaccurate answers, omit key information, or include irrelevant or redundant text, producing socially unacceptable or undesirable text even if the prompt is not offensive.

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and has established policies and practices for AI development. When developers download or use this model according to the terms of service, they should work with their internal model team to ensure it meets industry requirements and addresses potential product misuse. Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en - us/support/submit - security - vulnerability/).

References

• Compact Language Models via Pruning and Knowledge Distillation
• LLM Pruning and Distillation in Practice: The Minitron Approach

📄 License

This model is released under the [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia - open - model - license - agreement - june - 2024.pdf).