T5 Base Dutch

🚀 t5-base-dutch

This is a Dutch T5 model pre - trained during the Hugging Face community week, aiming to provide powerful natural language processing capabilities for Dutch.

🚀 Quick Start

This t5-base-dutch model was created by Yeb Havinga & Dat Nguyen during the Hugging Face community week, organized by HuggingFace with TPU usage sponsored by Google, for the project Pre - train T5 from scratch in Dutch.

You can also check out the fine - tuned t5-base-dutch-demo model and the demo application Netherformer 📰, which are based on this model.

5 Jan 2022: Model updated. Evaluation accuracy increased from 0.64 to 0.70.

11 Jan 2022: See also yhavinga/t5-v1.1-base-dutch-cased with eval acc 0.78

✨ Features

• Parameter Scale: This t5 model has 222M parameters.
• Pre - training Details: It was pre - trained with masked language modeling (denoise token span corruption) objective on the dataset mc4_nl_cleaned config full for 1 epoch(s) and a duration of 2d9h, with a sequence length of 512, batch size 128 and 527500 total steps (35B tokens). Pre - training evaluation loss and accuracy are 1,38 and 0,70.

📚 Documentation

Tokenizer

The model uses a cased SentencePiece tokenizer configured with the Nmt, NFKC, Replace multi - space to single - space normalizers and has 32003 tokens. It was trained on Dutch mc4 with scripts from the Huggingface Transformers [Flax examples](https://github.com/huggingface/transformers/tree/master/examples/flax/language - modeling). See ./raw/main/tokenizer.json for details.

Dataset(s)

All models listed below are pre - trained on cleaned Dutch mC4, which is the original mC4, except:

• Documents that contained words from a selection of the Dutch and English List of Dirty Naught Obscene and Otherwise Bad Words are removed.
• Sentences with less than 3 words are removed.
• Sentences with a word of more than 1000 characters are removed.
• Documents with less than 5 sentences are removed.
• Documents with "javascript", "lorum ipsum", "terms of use", "privacy policy", "cookie policy", "uses cookies", "use of cookies", "use cookies", "elementen ontbreken", "deze printversie" are removed.

The Dutch and English models are pre - trained on a 50/50% mix of Dutch mC4 and English C4. The translation models are fine - tuned on CCMatrix.

Dutch T5 Models

Three types of [Dutch T5 models have been trained (blog)](https://huggingface.co/spaces/yhavinga/pre - training - dutch - t5 - models). t5 - base - dutch is the only model with an original T5 config. The other model types t5 - v1.1 and t5 - eff have gated - relu instead of relu as activation function, and trained with a drop - out of 0.0 unless training would diverge (t5 - v1.1 - large - dutch - cased). The T5 - eff models are models that differ in their number of layers.

📄 License

This project is licensed under the apache - 2.0 license.