# Masked Language Prediction

Pile-T5 Base is an encoder-decoder model trained on The Pile dataset using the T5x library, trained for 2 million steps with MLM objective, approximately 2 trillion tokens.

SERENGETI is a large-scale multilingual pre-trained model covering 517 African languages and dialects, focusing on bridging the gap in technological resources for African languages.

A lightweight BERT model pretrained using knowledge distillation techniques, with a hidden layer dimension of 256 and 4 attention heads, suitable for masked language modeling tasks.