Mgpt

🚀 mGPT

mGPT is a pre-trained model on massive multilingual data, which can be used for text generation and downstream task adaptation.

🚀 Quick Start

mGPT is pre-trained on the mC4 dataset using a causal language modeling objective. It was introduced in this paper and first released on this page.

✨ Features

Model description

mGPT is a Transformer-based model which pre-trained on massive multilingual data covering over 101 languages. Similar to GPT-2, It was pre-trained on the raw texts only, with no human labeling. We use the same tokenization and vocabulary as the mT5 model.

Intended uses

You can use the raw model for text generation or using prompts for adapting it to a downstream task.

📦 Installation

No installation steps are provided in the original README, so this section is skipped.

💻 Usage Examples

Basic Usage

from transformers import MT5Tokenizer, GPT2LMHeadModel, TextGenerationPipeline

tokenizer = MT5Tokenizer.from_pretrained("THUMT/mGPT")
model = GPT2LMHeadModel.from_pretrained("THUMT/mGPT")

pipeline = TextGenerationPipeline(model=model, tokenizer=tokenizer)
text = "Replace me by any text you'd like."
text = pipeline(text, do_sample=True,  max_length=1024)[0]["generated_text"]

Advanced Usage

No advanced usage code examples are provided in the original README, so this part is skipped.

📚 Documentation

Preprocessing

The texts are tokenized using sentencepiece and a vocabulary size of 250,100. The inputs are sequences of 1,024 consecutive tokens. We use <extra_id_0> to separate lines in a document.

BibTeX entry and citation info

@misc{tan2021msp,
    title={MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators},
    author={Zhixing Tan and Xiangwen Zhang and Shuo Wang and Yang Liu},
    year={2021},
    eprint={2110.06609},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

🔧 Technical Details

No specific technical details (more than 50 - word description) are provided in the original README, so this section is skipped.

📄 License

No license information is provided in the original README, so this section is skipped.