Japanese Gpt2 Medium

🚀 Japanese GPT-2 Medium

This repository offers a medium-sized Japanese GPT-2 model. It aims to provide a powerful tool for Japanese text generation tasks, leveraging the capabilities of the GPT-2 architecture.

🚀 Quick Start

Here's how you can use the model:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("rinna/japanese-gpt2-medium", use_fast=False)
tokenizer.do_lower_case = True  # due to some bug of tokenizer config loading

model = AutoModelForCausalLM.from_pretrained("rinna/japanese-gpt2-medium")

✨ Features

• Model Architecture: A 24 - layer, 1024 - hidden - size transformer - based language model.
• Tokenization: The model uses a sentencepiece - based tokenizer. The vocabulary was trained on the Japanese Wikipedia using the official sentencepiece training script.

📦 Installation

The installation process is mainly about setting up the necessary Python libraries. You can install the transformers library via pip install transformers to use the model.

📚 Documentation

Training

The model was trained on [Japanese CC - 100](http://data.statmt.org/cc - 100/ja.txt.xz) and Japanese Wikipedia to optimize a traditional language modelling objective on 8*V100 GPUs for around 30 days. It reaches around 18 perplexity on a chosen validation set from the same data.

Release Date

April 7, 2021 (Updated: August 25, 2021)

How to Cite

@misc{rinna-japanese-gpt2-medium,
    title = {rinna/japanese-gpt2-medium},
    author = {Zhao, Tianyu and Sawada, Kei},
    url = {https://huggingface.co/rinna/japanese-gpt2-medium}
}

@inproceedings{sawada2024release,
    title = {Release of Pre-Trained Models for the {J}apanese Language},
    author = {Sawada, Kei and Zhao, Tianyu and Shing, Makoto and Mitsui, Kentaro and Kaga, Akio and Hono, Yukiya and Wakatsuki, Toshiaki and Mitsuda, Koh},
    booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
    month = {5},
    year = {2024},
    pages = {13898--13905},
    url = {https://aclanthology.org/2024.lrec-main.1213},
    note = {\url{https://arxiv.org/abs/2404.01657}}
}

📄 License

This project is released under The MIT license.

Additional Information