T5 Efficient Tiny Nl2

🚀 T5-Efficient-TINY-NL2 (Deep-Narrow version)

T5-Efficient-TINY-NL2 is a variant of Google's original T5, following the T5 model architecture. It's a pretrained-only checkpoint, released with the paper Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers by Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler. This model shows that a Deep-Narrow architecture can offer better downstream performance compared to other models with a similar parameter count.

✨ Features

The paper suggests that increasing the model depth (number of stacked transformer blocks) before other dimensions can lead to better Pareto-efficiency. Specifically, a deep and narrow model is generally more efficient than a base or large model. However, the relative gain in efficiency diminishes as more layers are added, converging at 32 - 36 layers.

📚 Documentation

Details model architecture

The t5-efficient-tiny-nl2 checkpoint is a Tiny model with nl set to 2. It has 11.9 million parameters, requiring ca. 47.61 MB of memory in full precision (fp32) or 23.81 MB in half precision (fp16 or bf16).

Here's a summary of the original T5 model architectures:

Abbreviations used:

If a model checkpoint has no specific el or dl, both encoder and decoder layers equal nl.

Pre - Training

The checkpoint was pretrained on the Colossal, Cleaned version of Common Crawl (C4) for 524288 steps using the span - based masked language modeling (MLM) objective.

Fine - Tuning

⚠️ Important Note

This model is a pretrained checkpoint and needs to be fine - tuned for practical use. It was pretrained in English, so it's only suitable for English NLP tasks.

You can follow these examples for fine - tuning:

PyTorch:

• Summarization
• [Question Answering](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question - answering/run_seq2seq_qa.py)
• [Text Classification](https://github.com/huggingface/transformers/tree/master/examples/pytorch/text - classification) - Note: You'll need to slightly adapt the training example for an encoder - decoder model.

Tensorflow:

• Summarization
• [Text Classification](https://github.com/huggingface/transformers/tree/master/examples/tensorflow/text - classification) - Note: Adapt the training example for an encoder - decoder model.

JAX/Flax:

• Summarization
• [Text Classification](https://github.com/huggingface/transformers/tree/master/examples/flax/text - classification) - Note: Adapt the training example for an encoder - decoder model.

More information

We highly recommend reading the original paper Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers for a deeper understanding. As mentioned in this issue, checkpoints with sh or skv architecture variations haven't been ported to Transformers due to limited practical use and lack of detailed description. These checkpoints are stored here and might be ported in the future.

📄 License

This project is licensed under the Apache - 2.0 license.