Llm Jp 3.1 1.8b Instruct4

🚀 llm-jp-3.1-1.8b-instruct4

LLM-jp-3.1 is a series of large language models developed by the Research and Development Center for Large Language Models at the National Institute of Informatics. It builds upon the LLM-jp-3 series and incorporates mid-training (instruction pre-training), significantly enhancing instruction-following capabilities compared to the original LLM-jp-3 models. This repository provides the llm-jp-3.1-1.8b-instruct4 model.

🚀 Quick Start

This section will guide you through the basic steps to get started with the llm-jp-3.1-1.8b-instruct4 model.

✨ Features

• Enhanced Instruction-Following: Incorporates mid-training (instruction pre-training) to significantly improve the ability to follow instructions.
• Multi-Language Support: Trained on a diverse set of languages, including Japanese, English, Code, Chinese, and Korean.
• Flexible Training: Supports pre-training, mid-training, and post-training, allowing for fine-tuning on specific tasks.

📦 Installation

Required Libraries and Their Versions

• torch>=2.3.0
• transformers>=4.40.1
• tokenizers>=0.19.1
• accelerate>=0.29.3
• flash-attn>=2.5.8

You can install these libraries using pip:

pip install torch>=2.3.0 transformers>=4.40.1 tokenizers>=0.19.1 accelerate>=0.29.3 flash-attn>=2.5.8

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("llm-jp/llm-jp-3.1-1.8b-instruct4")
model = AutoModelForCausalLM.from_pretrained("llm-jp/llm-jp-3.1-1.8b-instruct4", device_map="auto", torch_dtype=torch.bfloat16)
chat = [
    {"role": "system", "content": "以下は、タスクを説明する指示です。要求を適切に満たす応答を書きなさい。"},
    {"role": "user", "content": "自然言語処理とは何か"},
]
tokenized_input = tokenizer.apply_chat_template(chat, add_generation_prompt=True, tokenize=True, return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(
        tokenized_input,
        max_new_tokens=100,
        do_sample=True,
        top_p=0.95,
        temperature=0.7,
        repetition_penalty=1.05,
    )[0]
print(tokenizer.decode(output))

📚 Documentation

Model Details

Dense model

MoE model

Tokenizer

The tokenizer of this model is based on huggingface/tokenizers Unigram byte-fallback model. The vocabulary entries were converted from llm-jp-tokenizer v3.0. Please refer to README.md of llm-jp-tokenizer for details on the vocabulary construction procedure (the pure SentencePiece training does not reproduce our vocabulary).

Datasets

Pre-training

The models have been pre-trained using a blend of the following datasets:

Mid-training

In the LLM-jp-3.1 series, continuous pre-training was performed based on Instruction Pre-Training. Instruction Pre-Training enhances a model’s ability to follow instructions by continuing pre-training on a large collection of instruction–response pairs. Approximately 90B tokens of instruction–response data were prepared and mixed with the pre-training datasets, conducting continuous pre-training on a total of 400B tokens. Each model was initialized from existing checkpoints (llm-jp/llm-jp-3-1.8b, llm-jp/llm-jp-3-13b, and llm-jp/llm-jp-3-8x13b) and underwent continuous instruction pre-training. Since the LLM-jp-3 series was originally pre-trained on 2.1T tokens, the total pre-training token count amounts to 2.5T tokens.

Details of this training process will be released in a forthcoming paper. The instruction–response dataset used for this training will also be made publicly available.

Post-training

The pre-trained checkpoint was fine-tuned with supervised fine-tuning and further aligned with Direct Preference Optimization.

Supervised Fine-tuning

The datasets used for supervised fine-tuning are as follows:

Direct Preference Optimization

For Direct Preference Optimization (DPO), rejection sampling was adopted. Prompts were sampled from the dataset used in SFT, and multiple responses were generated for each prompt. These responses were then scored (by Qwen/Qwen2.5-32B-Instruct), and DPO was performed by treating high-scoring responses as positive examples and low-scoring responses as negative examples.

DPO was conducted in two stages. In the second stage, ac-self-inst, a Japanese preference dataset focused on safety, was additionally used.

Evaluation

MT Bench (Japanese and English)

The models were evaluated using gpt-4o-2024-08-06. The scores represent the average values obtained from three rounds of inference and evaluation. For more details, please refer to the codes.

AnswerCarefully-Eval

AnswerCarefully-Eval assesses the safety of Japanese language model outputs using the LLM-as-a-Judge approach, based on the test set from llm-jp/AnswerCarefully. The models were evaluated using gpt-4o-2024-08-06. The scores represent the average values obtained from three rounds of inference and evaluation. For more details, please refer to the codes.

🔧 Technical Details

The models in the LLM-jp-3.1 series are based on the Transformer architecture. They are trained on a large corpus of text data from multiple languages, including Japanese, English, Code, Chinese, and Korean. The mid-training process, Instruction Pre-Training, enhances the models' ability to follow instructions by continuing pre-training on a large collection of instruction–response pairs. The post-training process includes supervised fine-tuning and Direct Preference Optimization to further align the models with human preferences.

📄 License

This project is licensed under the Apache License, Version 2.0.

📬 Send Questions to

If you have any questions or need further assistance, please contact us at llm-jp(at)nii.ac.jp.

👥 Model Card Authors

The names are listed in alphabetical order.

Hirokazu Kiyomaru and Takashi Kodama.

⚠️ Important Note

The models released here are in the early stages of our research and development and have not been tuned to ensure outputs align with human intent and safety considerations.