Dorna Llama3 8B Instruct Quantized4Bit

🚀 Dorna-Llama3-8B-Instruct-Quantized4Bit

This project provides a 4-bit quantized version of the Dorna-Llama3-8B-Instruct model, aiming to optimize memory usage. The Dorna model, a decoder-only architecture, is specifically fine - tuned on Persian data. It also integrates Flash Attention 2 for accelerated inference.

🚀 Quick Start

You can run conversational inference using the Transformers Auto classes with the generate() function. Here is an example:

import torch
import transformers
from transformers import AutoTokenizer, AutoModelForCausalLM

model_path = "amirMohammadi/Dorna-Llama3-8B-Instruct-Quantized4Bit"

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "system",
     "content": "You are a helpful Persian assistant. Please answer questions in the asked language."},
    {"role": "user", "content": "اصفهان بزرگ تر است یا قم؟"},
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
)
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))

✨ Features

• Reduced Memory Usage: 4-bit quantization lowers memory requirements.
• Faster Inference: Flash Attention 2 speeds up processing.
• Easy Deployment: No need for additional libraries like LlamaCPP or Candle.
• Ready to Use: Compatible with Langchain, Haystack, LlamaIndex 2, and more.
• Google Colab Friendly: Can run on Google Colab free tier with T4 GPU (less than 15 GB of GPU RAM).

📚 Documentation

Evaluation of Non - Quantized version

This model is evaluated on questions across various tasks, including Boolean Questions, Code Generation, Long Response, Math, News QA, Paraphrasing, General Knowledge, and Summarization. Most categories typically have two main difficulty levels: Hard and Easy.

Both human evaluation and automatic evaluation (with GPT - 4 as the judge) are performed. In both tables, Dorna - 8B - it is used as an abbreviated form of Dorna - Llama3 - 8B - Instruct.

Overall human evaluation results

Category - based human evaluation results

Win/Lose/Tie % is reported for each category.

Automatic evaluation results

📄 License

This model is under the llama3 license.

📞 Contact us

If you have any questions regarding this model, you can reach us via the community on Hugging Face.