Glot500 Base

🚀 Glot500 (base-sized model)

The Glot500 model (Glot500-m) is pre-trained on over 500 languages using a masked language modeling (MLM) objective. It was introduced in this paper (ACL 2023) and first released in this repository. This model offers a powerful solution for multilingual language processing tasks.

🚀 Quick Start

Model Information

✨ Features

• Pre-trained on 500+ languages with a masked language modeling objective.
• Can be used directly with a pipeline for masked language modeling.
• Allows users to extract features of given text in PyTorch.

💻 Usage Examples

Basic Usage

You can use this model directly with a pipeline for masked language modeling:

>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='cis-lmu/glot500-base')
>>> unmasker("Hello I'm a <mask> model.")

Advanced Usage

Here is how to use this model to get the features of a given text in PyTorch:

>>> from transformers import AutoTokenizer, AutoModelForMaskedLM

>>> tokenizer = AutoTokenizer.from_pretrained('cis-lmu/glot500-base')
>>> model = AutoModelForMaskedLM.from_pretrained("cis-lmu/glot500-base")

>>> # prepare input
>>> text = "Replace me by any text you'd like."
>>> encoded_input = tokenizer(text, return_tensors='pt')

>>> # forward pass
>>> output = model(**encoded_input)

📚 Documentation

BibTeX entry and citation info

@article{imanigooghari-etal-2023-glot500,
  title={Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages},
  author={ImaniGooghari, Ayyoob  and Lin, Peiqin  and Kargaran, Amir Hossein  and Severini, Silvia  and Jalili Sabet, Masoud  and Kassner, Nora  and Ma, Chunlan  and Schmid, Helmut  and Martins, Andr{\'e}  and Yvon, Fran{\c{c}}ois  and Sch{\"u}tze, Hinrich},
  journal={arXiv preprint arXiv:2305.12182},
  year={2023}
}

📄 License

This project is licensed under the apache-2.0 license.