Emma 500 Llama2 7b

🚀 EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models

EMMA-500 is a cutting - edge multilingual language model. It aims to enhance language representation, especially for low - resource languages. This is achieved through continual pre - training on the Llama 2 7B architecture. By leveraging the MaLA Corpus, which covers over 500 languages and 74 billion tokens, EMMA-500 excels in various multilingual tasks such as commonsense reasoning, machine translation, open - ended generation, and text classification. It outperforms other Llama 2 - based models in diverse multilingual scenarios while remaining robust in specialized tasks.

📄 License

The license used for this project is llama2.

📚 Documentation

Model Details

Data Access

• MaLA Corpus
• PolyWrite Benchmark

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "MaLA-LM/emma-500-llama2-7b"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

input_text = "Once upon a time"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

📊 Model Performance

EMMA-500 was evaluated across multiple benchmarks and tasks, showing the following results:

• Lowest negative log - likelihood in intrinsic evaluations.
• Significant improvements in commonsense reasoning, machine translation, and open - ended generation.
• Outperformed all Llama 2 - based models in text classification and natural language inference.
• Enhanced performance in code generation and machine reading comprehension (MRC).

However, challenges remain in low - resource languages, where the model tends to have higher Self - BLEU scores, indicating reduced output diversity.

📖 Citation

@article{ji2024emma500enhancingmassivelymultilingual,
      title={{EMMA}-500: Enhancing Massively Multilingual Adaptation of Large Language Models}, 
      author={Shaoxiong Ji and Zihao Li and Indraneil Paul and Jaakko Paavola and Peiqin Lin and Pinzhen Chen and Dayyán O'Brien and Hengyu Luo and Hinrich Schütze and Jörg Tiedemann and Barry Haddow},
      year={2024},
      journal={arXiv preprint 2409.17892},
      url={https://arxiv.org/abs/2409.17892}, 
}

🙏 Acknowledgements

We would like to express our gratitude to the language communities and contributors who assisted in sourcing, cleaning, and validating the diverse data used in the MaLA Corpus. Their efforts are of great value in supporting linguistic diversity in AI research.

This work is carried out by researchers at [Helsinki - NLP](https://huggingface.co/Helsinki - NLP) in collaboration with partners from TU Darmstadt, the University of Edinburgh, and LMU Munich. It is funded by [HPLT](https://hplt - project.org) and [UTTER](https://he - utter.eu).