Llama 3.1 8b ITA

🚀 Llama-3.1-8b-ITA

Llama-3.1-8b-ITA is a specialized model based on Meta-Llama-3.1-8B-Instruct, tailored for the Italian language, with performance evaluated on multiple text generation tasks.

🚀 Quick Start

Llama-3.1-8b-ITA is a model specialized for the Italian language, based on the Meta-Llama-3.1-8B-Instruct base model.

✨ Features

• Model Architecture:
  • Base Model: Meta-Llama-3.1-8B-Instruct
  • Specialization: Italian Language

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

MODEL_NAME = "DeepMount00/Llama-3.1-8b-Ita"

model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, torch_dtype=torch.bfloat16).eval()
model.to(device)
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

def generate_answer(prompt):
    messages = [
        {"role": "user", "content": prompt},
    ]
    model_inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(device)
    generated_ids = model.generate(model_inputs, max_new_tokens=200, do_sample=True,
                                          temperature=0.001)
    decoded = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
    return decoded[0]

prompt = "Come si apre un file json in python?"
answer = generate_answer(prompt)
print(answer)

📚 Documentation

Model Evaluation Results

Detailed results can be found here

Evaluation Metrics

Model Index Details

• Name: Llama-3.1-8b-ITA
• Results:
  • Task: Text Generation on IFEval (0-Shot)
    • Dataset: HuggingFaceH4/ifeval
    • Metrics:
      • strict accuracy: 79.17
    • Source: Open LLM Leaderboard
  • Task: Text Generation on BBH (3-Shot)
    • Dataset: BBH
    • Metrics:
      • normalized accuracy: 30.93
    • Source: Open LLM Leaderboard
  • Task: Text Generation on MATH Lvl 5 (4-Shot)
    • Dataset: hendrycks/competition_math
    • Metrics:
      • exact match: 10.88
    • Source: Open LLM Leaderboard
  • Task: Text Generation on GPQA (0-shot)
    • Dataset: Idavidrein/gpqa
    • Metrics:
      • acc_norm: 5.03
    • Source: Open LLM Leaderboard
  • Task: Text Generation on MuSR (0-shot)
    • Dataset: TAUR-Lab/MuSR
    • Metrics:
      • acc_norm: 11.4
    • Source: Open LLM Leaderboard
  • Task: Text Generation on MMLU-PRO (5-shot)
    • Dataset: TIGER-Lab/MMLU-Pro
    • Metrics:
      • accuracy: 31.96
    • Source: Open LLM Leaderboard

👨‍💻 Developer

[Michele Montebovi]