T5 V1.1 Base Dutch Cnn Test

🚀 T5 v1.1 Base finetuned for CNN news summarization in Dutch 🇳🇱

This model is a fine - tuned version of [t5 - v1.1 - base - dutch - cased](https://huggingface.co/yhavinga/t5 - v1.1 - base - dutch - cased) on CNN Dailymail NL.

For a demo of the Dutch CNN summarization models, head over to the Hugging Face Spaces for the [Netherformer 📰](https://huggingface.co/spaces/flax - community/netherformer) example application!

The Rouge scores for this model are listed below.

🚀 Quick Start

This README provides a detailed introduction to the T5 v1.1 Base model fine - tuned for Dutch CNN news summarization, including tokenizer, dataset, model details, and Rouge scores.

✨ Features

Tokenizer

• A SentencePiece tokenizer was trained from scratch for Dutch on mC4 nl cleaned, using scripts from the Huggingface Transformers [Flax examples](https://github.com/huggingface/transformers/tree/master/examples/flax/language - modeling).

Dataset

All the models below are trained on the full configuration (39B tokens) of cleaned Dutch mC4. The cleaning process involves:

• Removing documents that contain words from a selection of the Dutch and English [List of Dirty Naught Obscene and Otherwise Bad Words](https://github.com/LDNOOBW/List - of - Dirty - Naughty - Obscene - and - Otherwise - Bad - Words).
• Removing sentences with less than 3 words.
• Removing sentences with a word of more than 1000 characters.
• Removing documents with less than 5 sentences.
• Removing documents with "javascript", "lorum ipsum", "terms of use", "privacy policy", "cookie policy", "uses cookies", "use of cookies", "use cookies", "elementen ontbreken", "deze printversie".

Models

TL;DR: [yhavinga/t5 - v1.1 - base - dutch - cased](https://huggingface.co/yhavinga/t5 - v1.1 - base - dutch - cased) is the best model.

• yhavinga/t5 - base - dutch is a re - trained Dutch T5 base v1.0 model from the summer 2021 Flax/Jax community week. Its accuracy was improved from 0.64 to 0.70.
• The two T5 v1.1 base models are an uncased and cased version of t5 - v1.1 - base, pre - trained from scratch on Dutch, with a tokenizer also trained from scratch. The t5 v1.1 models are slightly different from the t5 models, and the base models are trained with a dropout of 0.0. For fine - tuning, it is intended to set this back to 0.1.
• The large cased model is a pre - trained Dutch version of t5 - v1.1 - large. Training t5 - v1.1 - large was difficult. Without dropout regularization, the training would diverge at a certain point. With dropout, training went better, but much slower than training the t5 - model. At some point, the convergence was too slow to continue training. The latest checkpoint, training scripts, and metrics are available for reference. For actual fine - tuning, the cased base model is probably a better choice.

The cased t5 - v1.1 Dutch models were fine - tuned on summarizing the CNN Daily Mail dataset.

📚 Documentation

The Rouge scores for the model yhavinga/t5 - v1.1 - base - dutch - cnn - test on the test split of ml6team/cnn_dailymail_nl are as follows:

📄 License

This project is licensed under the Apache - 2.0 license.

👏 Acknowledgements

This project would not have been possible without the generous compute resources provided by Google through the TPU Research Cloud. The HuggingFace 🤗 ecosystem was also crucial in many, if not all, parts of the training. The following repositories were helpful in setting up the TPU - VM and training the models:

• [Gsarti's Pretrain and Fine - tune a T5 model with Flax on GCP](https://github.com/gsarti/t5 - flax - gcp)
• [HUggingFace Flax MLM examples](https://github.com/huggingface/transformers/tree/master/examples/flax/language - modeling)
• [Flax/Jax Community week t5 - base - dutch](https://huggingface.co/flax - community/t5 - base - dutch)

Created by [Yeb Havinga](https://www.linkedin.com/in/yeb - havinga - 86530825/)