Eurollm 1.7B Instruct

🚀 EuroLLM-1.7B-Instruct

This is the first instruction-tuned model in the EuroLLM series. It can understand and generate text in multiple languages, offering strong performance in machine translation and general benchmarks.

🚀 Quick Start

To run the EuroLLM-1.7B-Instruct model, you can use the following code:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "utter-project/EuroLLM-1.7B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

text = '<|im_start|>system\n<|im_end|>\n<|im_start|>user\nTranslate the following English source text to Portuguese:\nEnglish: I am a language model for european languages. \nPortuguese: <|im_end|>\n<|im_start|>assistant\n'

inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

✨ Features

• Multilingual Support: Capable of handling 34 languages, including most European Union languages and some additional relevant ones.
• Instruction Tuned: Fine-tuned on EuroBlocks for better general instruction-following and machine translation capabilities.
• Strong Performance: Outperforms Gemma-2B in machine translation and is competitive with Gemma-7B.

📦 Installation

The installation mainly involves using the transformers library. You can install it via the following command:

pip install transformers

📚 Documentation

Model Details

The EuroLLM project aims to create a suite of LLMs that can understand and generate text in all European Union languages and some other relevant languages. EuroLLM-1.7B is a 1.7B parameter model trained on 4 trillion tokens from multiple data sources, including web data, parallel data, and high-quality datasets. EuroLLM-1.7B-Instruct is further instruction-tuned on EuroBlocks.

Model Description

EuroLLM uses a standard, dense Transformer architecture:

• Grouped Query Attention (GQA): With 8 key-value heads to increase inference speed while maintaining downstream performance.
• Pre-layer Normalization: Using RMSNorm for improved training stability and faster computation.
• SwiGLU Activation Function: Proven to yield good results on downstream tasks.
• Rotary Positional Embeddings (RoPE): Applied in every layer for good performance and context length extension.

Here is a summary of the model hyper-parameters:

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "utter-project/EuroLLM-1.7B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

text = '<|im_start|>system\n<|im_end|>\n<|im_start|>user\nTranslate the following English source text to Portuguese:\nEnglish: I am a language model for european languages. \nPortuguese: <|im_end|>\n<|im_start|>assistant\n'

inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

🔧 Technical Details

For pre-training, 256 Nvidia H100 GPUs of the Marenostrum 5 supercomputer were used. The model was trained with a constant batch size of 3,072 sequences (approximately 12 million tokens), using the Adam optimizer and BF16 precision.

📄 License

This model is licensed under the Apache License 2.0.

Results

Machine Translation

We evaluated EuroLLM-1.7B-Instruct on several machine translation benchmarks, including FLORES-200, WMT-23, and WMT-24, and compared it with Gemma-2B and Gemma-7B (also instruction tuned on EuroBlocks). The results show that EuroLLM-1.7B is substantially better than Gemma-2B in machine translation and competitive with Gemma-7B.

FLORES-200

WMT-23

WMT-24

General Benchmarks

We also compared EuroLLM-1.7B with TinyLlama-v1.1 and Gemma-2B on 3 general benchmarks: Arc Challenge and Hellaswag. For non-English languages, we used the Okapi datasets. Results show that EuroLLM-1.7B is superior to TinyLlama-v1.1 and similar to Gemma-2B on Hellaswag but worse on Arc Challenge. This may be due to the lower number of parameters of EuroLLM-1.7B (1.133B non-embedding parameters compared to 1.981B).

Arc Challenge

Hellaswag

Bias, Risks, and Limitations

EuroLLM-1.7B-Instruct has not been aligned to human preferences, so the model may generate problematic outputs (e.g., hallucinations, harmful content, or false statements).

Paper

Paper: EuroLLM: Multilingual Language Models for Europe