Llama 3.1 Tango 70b Bnb 4b

🚀 Tango-70B-Instruct

Tango-70B-Instruct is a large language model trained to enhance regional Spanish speech performance.

🚀 Quick Start

Tango-70B-Instruct can be used via the HuggingFace Transformers library. You'll need 2 or more 80GB GPUs (NVIDIA Ampere or newer) and at least 150GB of free disk space for the download.

This code has been tested on Transformers v4.44.0, torch v2.4.0, and 2 A100 80GB GPUs. Any setup that supports meta-llama/Llama-3.1-70B-Instruct should also support this model. If you encounter issues, you can try pip install -U transformers.

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

# Load base model and tokenizer
base_model_id = "nvidia/Llama-3.1-Nemotron-70B-Instruct-HF"
adapter_model_id = "sandbox-ai/Tango-70b"

# Create quantization config for 4-bit precision
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

# Load tokenizer from base model
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

# Load the base model with 4-bit quantization
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto",  # This will automatically handle model sharding
    trust_remote_code=True
)

# Load the PEFT adapter
model = PeftModel.from_pretrained(
    base_model,
    adapter_model_id,
    device_map="auto",  # This will automatically handle model sharding
)

hola_mundo = """
Bienvenido. 
Tu nombre es "Tango", sos la primer IA hecha en LatinoAmérica, basada en un Large Language Model de 70 billones de parámetros y creada en Argentina. 

Cuál es la importancia de hacer IA nativa en LatinoAmérica? qué beneficios trae haberte creado, en comparación a depender de las IAs creadas en USA, Francia o China?

"""

# Test prompt
messages = [
    {"role": "user", "content": hola_mundo}
]

# Format the input using the chat template
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

# Generate response with memory-efficient settings
with torch.inference_mode():
    outputs = model.generate(
        inputs,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.7,
        top_p=0.95,
        pad_token_id=tokenizer.eos_token_id,  # Set padding token
        attention_mask=torch.ones_like(inputs)  # Add attention mask
    )

# Decode and print the response
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

✨ Features

• Regional Spanish Enhancement: Tango-70B-Instruct is trained on a modified variation of spanish/-ir/messirve to improve regional Spanish speech performance.
• Multi - language Support: Supports both English and Spanish.

📦 Model Information

📚 Documentation

Model Overview

Tango-70B-Instruct is a large language model trained by sandbox-ai on a modified variation of spanish/-ir/messirve to improve the regional Spanish speech performance.

See details on the github repo

Terms of use

By accessing this model, you are agreeing to the LLama 3.1 terms and conditions of the license, acceptable use policy and Meta’s privacy policy

Evaluation Metrics

📄 License

This model is under the llama3.1 license. By accessing this model, you are agreeing to the LLama 3.1 terms and conditions of the license, acceptable use policy and Meta’s privacy policy