Sambalingo Arabic Chat 70B

🚀 SambaLingo-Arabic-Chat-70B

SambaLingo-Arabic-Chat-70B is a human-aligned chat model trained in Arabic and English, offering high - quality language interaction capabilities.

🚀 Quick Start

Loading Model With Hugging Face

Please make sure to set use_fast=False when loading the tokenizer.

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("sambanovasystems/SambaLingo-Arabic-Chat-70B", use_fast=False)
model = AutoModelForCausalLM.from_pretrained("sambanovasystems/SambaLingo-Arabic-Chat-70B", device_map="auto", torch_dtype="auto")

Interacting With Model Pipeline

Please make sure to set use_fast=False when loading the tokenizer.

from transformers import pipeline
pipe = pipeline("text-generation", model="sambanovasystems/SambaLingo-Arabic-Chat-70B", device_map="auto", use_fast=False)
messages = [
                {"role": "user", "content": {YOUR_QUESTION}},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt)[0]
outputs = outputs["generated_text"]

Suggested Inference Parameters

• Temperature: 0.8
• Repetition penalty: 1.0
• Top - p: 0.9

Prompting Guidelines

To prompt this model, please use the following chat template:

<|user|>\n{question}</s>\n<|assistant|>\n

Example Prompts and Generations

<|user|>
كيف تصف الالوان لشخص ولد أعمى</s>
<|assistant|>
ليس لدي القدرة على الرؤية، ولكن يمكنني تقديم وصف للألوان بناءً على ما تعلمته من البيانات.\n\nبالنسبة لشخص ولد أعمى، قد يكون من الصعب وصف الألوان لأنها لم تختبرها من قبل. ومع ذلك، يمكننا استخدام الاستعارات والمقارنات لمساعدة هذا الشخص على فهم الألوان. على سبيل المثال، يمكننا وصف اللون الأحمر بأنه دافئ وحيوي، مثل لهب النار أو ثمرة الفراولة الناضجة. يمكننا وصف اللون الأزرق بأنه هادئ ومهدئ، مثل السماء أو المحيط. يمكننا وصف اللون الأخضر بأنه منعش وطبيعي، مثل أوراق الشجر أو العشب. يمكننا وصف اللون الأصفر بأنه مشرق ومبهج، مثل الشمس أو زهرة عباد الشمس. يمكننا وصف اللون الأرجواني بأنه ملكي وفاخر، مثل لون الجواهر أو غروب الشمس. في النهاية، قد يكون من الصعب وصف الألوان لشخص ولد أعمى، ولكن باستخدام الاستعارات والمقارنات، يمكننا مساعدتهم على فهم الفروق الدقيقة في الألوان بشكل أفضل.

✨ Features

SambaLingo-Arabic-Chat-70B is a language model developed by SambaNova Systems. It supports both Arabic and English, and is fine - tuned from Llama-2-70b. You can find more details in the paper and blog post.

🔧 Technical Details

The alignment phase follows the recipe for Zephyr-7B, and comprises two stages: supervised fine - tuning (SFT) and Direct Performance Optimization (DPO).

The SFT phase was done on the ultrachat_200k dataset mixed with the Google translated version of the ultrachat_200k dataset. It was trained for one epoch with global batch size 512 and max sequence length 2048 tokens. A linear decay learning rate of 2e - 5 and 10% warmup were used.

The DPO phase was done on the ultrafeedback dataset and [cai - conversation - harmless](https://huggingface.co/datasets/HuggingFaceH4/cai - conversation - harmless) dataset, mixed with 10% of the data Google translated. It was trained with global batch size 32 and for three epochs. A linear decay learning rate of 5e - 7, 10% warmup and β = 0.1 as the regularization factor for DPO were used.

The vocabulary of the base llama model was extended from 32,000 tokens to 57,000 tokens by adding up to 25,000 non - overlapping tokens from the new language.

📚 Documentation

Uses

Direct Use

Use of this model is governed by the Meta’s Llama 2 Community License Agreement. Please review and accept the license before downloading the model weights.

Out - of - Scope Use

SambaLingo should NOT be used for:

• Mission - critical applications
• Applications that involve the safety of others
• Making highly important decisions

Bias, Risks, and Limitations

Like all LLMs, SambaLingo has certain limitations:

• Hallucination: The model may sometimes generate responses that contain plausible - sounding but factually incorrect or irrelevant information.
• Code Switching: The model might unintentionally switch between languages or dialects within a single response, affecting the coherence and understandability of the output.
• Repetition: The model may produce repetitive phrases or sentences, leading to less engaging and informative responses.
• Coding and Math: The model's performance in generating accurate code or solving complex mathematical problems may be limited.
• Toxicity: The model could inadvertently generate responses containing inappropriate or harmful content.

📄 License

The use of this model is subject to the Meta’s Llama 2 Community License Agreement.

Acknowledgments

We extend our heartfelt gratitude to the open - source AI community; this endeavor would not have been possible without open source. SambaNova embraces the open - source community and aspires to actively contribute to this initiative.

We would like to give a special thanks to the following groups:

• Meta for open sourcing LLama 2 and open sourcing FLORES - 200 dataset
• Nguyen et al for open sourcing CulturaX dataset
• CohereAI for releasing AYA - 101 and open sourcing a multilingual instruction tuning dataset
• EleutherAI for their open source evaluation framework
• Hugging Face - H4 team for open source the zephyr training recipe and alignment handbook repo

Cite SambaLingo

@misc{csaki2024sambalingo,
      title={SambaLingo: Teaching Large Language Models New Languages}, 
      author={Zoltan Csaki and Bo Li and Jonathan Li and Qiantong Xu and Pian Pawakapan and Leon Zhang and Yun Du and Hengyu Zhao and Changran Hu and Urmish Thakker},
      year={2024},
      eprint={2404.05829},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}