Japanese Gpt Neox Small

🚀 Japanese GPT-NeoX Small

This repository offers a small-sized Japanese GPT-NeoX model, trained using code from EleutherAI/gpt-neox.

🚀 Quick Start

Installation

To use the model, you need to install the transformers library. You can install it using pip:

pip install transformers

Usage

from transformers import AutoTokenizer, GPTNeoXForCausalLM

tokenizer = AutoTokenizer.from_pretrained("rinna/japanese-gpt-neox-small", use_fast=False)
model = GPTNeoXForCausalLM.from_pretrained("rinna/japanese-gpt-neox-small")

✨ Features

• Small Size: A compact Japanese GPT-NeoX model.
• Prefix-Tuning Support: Comes with a toy prefix-tuning weight file for demonstration.
• FasterTransformer Compatibility: Supports inference with NVIDIA FasterTransformer since version 5.1.

📦 Installation

Install the necessary dependencies using pip:

pip install transformers

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, GPTNeoXForCausalLM

tokenizer = AutoTokenizer.from_pretrained("rinna/japanese-gpt-neox-small", use_fast=False)
model = GPTNeoXForCausalLM.from_pretrained("rinna/japanese-gpt-neox-small")

input_text = "こんにちは"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids
output = model.generate(input_ids)
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)

Advanced Usage with Prefix-Tuning

You can use the provided prefix-tuning weight file smileface_suffix.task0.weight to encourage the model to end generated sentences with a smiling face emoji 😃. Find the training/inference code for prefix-tuning at our Github repo prefix-tuning-gpt.

📚 Documentation

Model Architecture

The model is a 12-layer, 768-hidden-size transformer-based language model.

Training

The model was trained on Japanese CC-100, Japanese C4, and Japanese Wikipedia to optimize a traditional language modelling objective.

Tokenization

The model uses a sentencepiece-based tokenizer.

🔧 Technical Details

Prefix-Tuning

We release a prefix-tuning weight file named smileface_suffix.task0.weight for demonstration. The toy prefix-tuning weights here is trained to encourage the model to end every generated sentence with a smiling face emoji 😃.

Inference with FasterTransformer

After version 5.1, NVIDIA FasterTransformer now supports both inference for GPT-NeoX and a variety of soft prompts (including prefix-tuning). The released pretrained model and prefix weights in this repo have been verified to work with FasterTransformer 5.1.

📄 License

This project is licensed under The MIT license.

📅 Release Date

September 5, 2022

📝 How to Cite

@misc{rinna-japanese-gpt-neox-small,
    title = {rinna/japanese-gpt-neox-small},
    author = {Zhao, Tianyu and Sawada, Kei},
    url = {https://huggingface.co/rinna/japanese-gpt-neox-small}
}

@inproceedings{sawada2024release,
    title = {Release of Pre-Trained Models for the {J}apanese Language},
    author = {Sawada, Kei and Zhao, Tianyu and Shing, Makoto and Mitsui, Kentaro and Kaga, Akio and Hono, Yukiya and Wakatsuki, Toshiaki and Mitsuda, Koh},
    booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
    month = {5},
    year = {2024},
    pages = {13898--13905},
    url = {https://aclanthology.org/2024.lrec-main.1213},
    note = {\url{https://arxiv.org/abs/2404.01657}}
}