Eurollm 9B

🚀 EuroLLM-9B

A 9B parameter multilingual transformer LLM capable of handling a wide range of languages.

🚀 Quick Start

Here is a simple example of how to run the EuroLLM-9B model:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "utter-project/EuroLLM-9B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

text = "English: My name is EuroLLM. Portuguese:"

inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

✨ Features

• Multilingual Support: Capable of understanding and generating text in a wide range of languages, including all European Union languages and some additional relevant languages.
• Optimized Architecture: Uses grouped query attention (GQA) with 8 key - value heads, pre - layer normalization with RMSNorm, SwiGLU activation function, and rotary positional embeddings (RoPE).
• Large - scale Training: Trained on 4 trillion tokens across multiple data sources.

📦 Installation

The installation process mainly involves installing the transformers library. You can install it via pip:

pip install transformers

📚 Documentation

Model Details

The EuroLLM project aims to create a suite of LLMs that can understand and generate text in all European Union languages and some additional relevant languages. EuroLLM-9B is a 9B parameter model trained on 4 trillion tokens from various data sources, including web data, parallel data, and high - quality datasets. EuroLLM-9B-Instruct was further instruction tuned on EuroBlocks.

Model Description

EuroLLM uses a standard, dense Transformer architecture:

• Grouped query attention (GQA) with 8 key - value heads to increase inference speed while maintaining downstream performance.
• Pre - layer normalization with RMSNorm for improved training stability and faster computation.
• SwiGLU activation function for good results on downstream tasks.
• Rotary positional embeddings (RoPE) in every layer for good performance and context length extension.

Here is a summary of the model hyper - parameters:

Results

EU Languages

Table 1: Comparison of open - weight LLMs on multilingual benchmarks. The borda count corresponds to the average ranking of the models (see (Colombo et al., 2022)). For Arc - challenge, Hellaswag, and MMLU we are using Okapi datasets ([Lai et al., 2023](https://aclanthology.org/2023.emnlp - demo.28/)) which include 11 languages. For MMLU - Pro and MUSR we translate the English version with Tower (Alves et al., 2024) to 6 EU languages.
* As there are no public versions of the pre - trained models, we evaluated them using the post - trained versions.

EuroLLM-9B shows superior performance on multilingual tasks compared to other European - developed models and strong competitiveness with non - European models.

English

Table 2: Comparison of open - weight LLMs on English general benchmarks.
* As there are no public versions of the pre - trained models, we evaluated them using the post - trained versions.

EuroLLM shows strong performance on English tasks, surpassing most European - developed models and matching the performance of Mistral - 7B.

Bias, Risks, and Limitations

⚠️ Important Note

EuroLLM-9B has not been aligned to human preferences, so the model may generate problematic outputs (e.g., hallucinations, harmful content, or false statements).