Byt5 Small Historic English Span20

🚀 hmByT5 - Preliminary Language Models

Preliminary Historic Multilingual and Monolingual ByT5 Models, offering solutions for multilingual and monolingual language processing.

This project presents preliminary historic multilingual and monolingual ByT5 models. Currently, it covers the following language:

• English (British Library Corpus - Books)

More details can be found in our GitHub repository.

🚀 Quick Start

This section provides an overview of the hmByT5 model and its current language coverage. For more in - depth information, please refer to the subsequent sections.

📚 Documentation

Pretraining

We use the official JAX/FLAX example in Hugging Face Transformers to pretrain a ByT5 model on a single v3 - 8 TPU. Details about the training can be found here.

This model was trained with mean_noise_span_length = 20 for one epoch.

Mean Noise Span Length

The previously pretrained hmByT5 models "accidentally" use a mean noise span length of 3, because this value is the default one for T5. But the ByT5 paper mentions that using a length of 3 would make pretraining tasks too easy, and recommends a value of 20. Thus, we pretrained this model with mean_noise_span_length = 20 and fine - tuned it on the English AjMC dataset:

For comparison, the model using a length of 3 achieved 85.65 ± 1.21.

📄 License

This project is licensed under the MIT license.

💡 Acknowledgements

Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). Many Thanks for providing access to the TPUs ❤️