T5 Efficient Tiny Nl8

🚀 T5-Efficient-TINY-NL8 (Deep-Narrow version)

T5-Efficient-TINY-NL8 is a variation of Google's original T5, following the T5 model architecture. It's a pretrained-only checkpoint, released with the paper Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers by Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler. This model shows that a Deep-Narrow architecture can achieve better downstream performance compared to other models with similar parameter counts.

✨ Features

The paper suggests that a Deep-Narrow model architecture is more favorable for downstream performance. It recommends increasing the model's depth before other forms of scaling. A deep and narrow model is generally more efficient than the base model, and a deep base model can be more efficient than a large model. The relative gain of Pareto - efficiency diminishes as the number of layers increases, converging at 32 to 36 layers.

📚 Documentation

Details model architecture

This model checkpoint - t5-efficient-tiny-nl8 - is of model type Tiny with the following variations:

• nl is 8

It has 22.93 million parameters and requires ca. 91.74 MB of memory in full precision (fp32) or 45.87 MB of memory in half precision (fp16 or bf16).

The following table shows a summary of the original T5 model architectures:

The following table explains the abbreviations:

If a model checkpoint has no specific el or dl, both the number of encoder - and decoder layers correspond to nl.

Pre - Training

The checkpoint was pretrained on the Colossal, Cleaned version of Common Crawl (C4) for 524288 steps using the span - based masked language modeling (MLM) objective.

Fine - Tuning

⚠️ Important Note

This model is a pretrained checkpoint and has to be fine - tuned for practical usage. The checkpoint was pretrained in English and is therefore only useful for English NLP tasks.

You can follow one of the following examples on how to fine - tune the model:

PyTorch:

• Summarization
• Question Answering
• Text Classification - Note: You will have to slightly adapt the training example here to make it work with an encoder - decoder model.

Tensorflow:

• Summarization
• Text Classification - Note: You will have to slightly adapt the training example here to make it work with an encoder - decoder model.

JAX/Flax:

• Summarization
• Text Classification - Note: You will have to slightly adapt the training example here to make it work with an encoder - decoder model.

More information

We strongly recommend the reader to go carefully through the original paper Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers to get a more nuanced understanding of this model checkpoint. As explained in the following issue, checkpoints including the sh or skv model architecture variations have not been ported to Transformers as they are probably of limited practical usage and are lacking a more detailed description. Those checkpoints are kept here as they might be ported potentially in the future.

📄 License

This model is released under the apache - 2.0 license.