Gemma 2 Llama Swallow 9b It V0.1

🚀 Gemma-2-Llama-Swallow

The Gemma-2-Llama-Swallow series enhances language capabilities by continual pre - training on Gemma 2 models, excelling in both English and Japanese.

🚀 Quick Start

To get started with the Gemma-2-Llama-Swallow series, first install the necessary library:

pip install vllm

Then, you can use the following Python code to generate text:

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_name = "tokyotech-llm/Gemma-2-Llama-Swallow-27b-it-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model_name)
llm = LLM(
    model=model_name,
    tensor_parallel_size=1,
)

sampling_params = SamplingParams(
    temperature=0.6, top_p=0.9, max_tokens=512,
)


message = [
    {
        "role": "user",
        "content": "日本の春から夏の移り変わりについて教えてください",
    },
]
prompt = tokenizer.apply_chat_template(
    message, tokenize=False, add_generation_prompt=True
)

output = llm.generate(prompt, sampling_params)

print(output[0].outputs[0].text)

✨ Features

• Enhanced Language Capabilities: Gemma 2 Swallow retains English language capabilities while significantly enhancing Japanese language capabilities.
• Continual Pre - training: Built by continual pre - training on the gemma - 2 models using approximately 200 billion tokens from diverse sources.
• Instruction - Tuned Models: Instruction - tuned models are built by supervised fine - tuning (SFT) on synthetic data specially built for Japanese.

📦 Installation

pip install vllm

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_name = "tokyotech-llm/Gemma-2-Llama-Swallow-27b-it-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model_name)
llm = LLM(
    model=model_name,
    tensor_parallel_size=1,
)

sampling_params = SamplingParams(
    temperature=0.6, top_p=0.9, max_tokens=512,
)


message = [
    {
        "role": "user",
        "content": "日本の春から夏の移り変わりについて教えてください",
    },
]
prompt = tokenizer.apply_chat_template(
    message, tokenize=False, add_generation_prompt=True
)

output = llm.generate(prompt, sampling_params)

print(output[0].outputs[0].text)

📚 Documentation

Release History

• May 19, 2025: Released Gemma-2-Llama-Swallow-2b-pt-v0.1, Gemma-2-Llama-Swallow-9b-pt-v0.1, Gemma-2-Llama-Swallow-27b-pt-v0.1, Gemma-2-Llama-Swallow-2b-it-v0.1, Gemma-2-Llama-Swallow-9b-it-v0.1, and Gemma-2-Llama-Swallow-27b-it-v0.1.

Swallow Model Index

The website https://swallow-llm.github.io/ provides large language models developed by the Swallow team.

Model Details

Model Performance

MT - Bench JA

Japanese tasks

English tasks

Evaluation Benchmarks

The evaluation script can be found at swallow-llm/swallow-evaluation, tagged as v202411.

MT - Bench JA

We used Japanese MT - Bench to assess the capabilities of multi - turn dialogue with the following settings:

• Implementation: FastChat [Zheng+, 2023] (commit #e86e70d0)
• Question: Nejumi LLM - Leaderboard NEO, mtbench_ja_question_v4
• Reference Answer: A revised version of Nejumi LLM - Leaderboard NEO, mtbench_ja_referenceanswer_v2, in which we verified and corrected incorrect answers. This revised version has been released alongside swallow - evaluation Ver. 202411.
• Prompt for Judge: Nejumi LLM - Leaderboard NEO, mtbench_ja_prompt_v1
• Judge: gpt - 4o - 2024 - 08 - 06
• Scoring: Absolute scale normalized to a 0 - 1 range, averaged over five runs.

Japanese evaluation benchmarks

We used llm - jp - eval(v1.3.0), JP Language Model Evaluation Harness(commit #9b42d41) and Code Generation LM Evaluation Harness(commit #0261c52). The details are as follows:

• Multiple - choice question answering (JCommonsenseQA [Kurihara et al., 2022])
• Open - ended question answering (JEMHopQA [Ishii et al., 2024])
• Open - ended question answering (NIILC [Èñ¢Ê†π, 2003])
• Machine reading comprehension (JSQuAD [Kurihara et al., 2022])
• Automatic summarization (XL - Sum [Hasan et al., 2021])
• Machine translation (WMT2020 ja - en [Barrault et al., 2020])
• Machine translation (WMT2020 en - ja [Barrault et al., 2020])
• Mathematical reasoning (MGSM [Shi et al., 2023])
• Academic exams (JMMLU [Â∞π„Çâ, 2024])
• Code generation (JHumanEval [‰ΩêËó§„Çâ, 2024])

English evaluation benchmarks

We used the Language Model Evaluation Harness(v.0.4.2) and Code Generation LM Evaluation Harness(commit #0261c52). The details are as follows:

• Multiple - choice question answering (OpenBookQA [Mihaylov et al., 2018])
• Open - ended question answering (TriviaQA [Joshi et al., 2017])
• Machine reading comprehension (SQuAD2 [Rajpurkar et al., 2018])
• Commonsense reasoning (XWINO [Tikhonov and Ryabinin, 2021])
• Natural language inference (HellaSwag [Zellers et al., 2019])
• Mathematical reasoning (GSM8K [Cobbe et al., 2021])
• Mathematical reasoning (MATH [Hendrycks et al., 2022][Lightman et al., 2024])
• Reasoning (BBH (BIG - Bench - Hard) [Suzgun et al., 2023])
• Academic exams (MMLU [Hendrycks et al., 2021])
• Code generation (HumanEval [Chen et al., 2021])

Training Datasets

Instruction Tuning

The following datasets were used for the instruction tuning:

• Gemma - 2 - LMSYS - Chat - 1M - Synth
  • Multi - turn Japanese instruction dataset synthesized and derived from lmsys - chat - 1m [Zhang+, ICLR24]).
  • First - turn user instructions were translated into Japanese via DeepL (machine translation), and assistant responses were generated using gemma - 2 - 27b - it. The same model, i.e., gemma - 2 - 27b - it served as a judge for rejection sampling (n = 6).
  • Second - turn user instructions and responses were synthesized using gemma - 2 - 27b - it. The same model scores the quality of the second - turn response with a range of 1 - 10. Second - turn responses with scores lower than 9 were rejected, along with their corresponding instructions.
    Conversations containing personally identifiable information (PII) and template - based user instructions were removed. Duplicate instructions were removed.
• Swallow - Magpie - Ultra - v0.1
  • A Japanese variant of the filtered - magpie - ultra - en dataset, translated into Japanese by gemma - 2 - 27b - it.
• Swallow - Gemma - Magpie - v0.1
  • A Japanese synthetic instruction tuning dataset from scratch, generated by gemma - 2 - 27b - it. User instructions were created with prompts specific to each topic, and assistant responses were generated for these instructions.
  • The conversations were heuristically filtered for quality and length. Then, gemma - 2 - 27b - it was applied to score the quality of each of the conversation with a range of 1 - 10. Conversations with scores <= 7 were rejected.

🔧 Technical Details

The Gemma-2-Llama-Swallow series was built by continual pre - training on the gemma - 2 models. Approximately 200 billion tokens were sampled from a large Japanese web corpus (Swallow Corpus Version 2), Japanese and English Wikipedia articles, and mathematical and coding contents, etc (see the Training Datasets section of the base model) for continual pre - training. The instruction - tuned models (it) were built by supervised fine - tuning (SFT) on the synthetic data specially built for Japanese.

📄 License

• gemma
• llama3.3

Risks and Limitations

The models released here are still in the early stages of our research and development and have not been tuned to ensure outputs align with human intent and safety considerations.

Acknowledgements

We thank Google DeepMind for releasing Gemma 2 under a generous open license.

We received various support, including:

• AIST project: "Research and Development of Foundation Models for Generative AI in the Physical Domain"
• NEDO project: "Development of Artificial Intelligence Application Technology to Support"