T5 Efficient Small El2

🚀 T5-Efficient-SMALL-EL2 (Deep-Narrow version)

T5-Efficient-SMALL-EL2 is a variant of Google's original T5, following the T5 model architecture. It's a pretrained-only checkpoint, released with the paper Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers by Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler.

In essence, the paper shows that a Deep-Narrow model architecture performs better in downstream tasks than other architectures with a similar number of parameters.

Here's a quote from the paper:

We generally recommend a DeepNarrow strategy where the model’s depth is preferentially increased before considering any other forms of uniform scaling across other dimensions. This is largely due to how much depth influences the Pareto-frontier as shown in earlier sections of the paper. Specifically, a tall small (deep and narrow) model is generally more efficient compared to the base model. Likewise, a tall base model might also generally more efficient compared to a large model. We generally find that, regardless of size, even if absolute performance might increase as we continue to stack layers, the relative gain of Pareto-efficiency diminishes as we increase the layers, converging at 32 to 36 layers. Finally, we note that our notion of efficiency here relates to any one compute dimension, i.e., params, FLOPs or throughput (speed). We report all three key efficiency metrics (number of params, FLOPS and speed) and leave this decision to the practitioner to decide which compute dimension to consider.

More precisely, model depth is the number of sequentially stacked transformer blocks. A sequence of word embeddings is processed sequentially by each transformer block.

📚 Documentation

✨ Features

• It's a variation of Google's original T5, following the T5 model architecture.
• A pretrained-only checkpoint, released with a relevant research paper.
• The Deep-Narrow architecture shows better downstream performance.

📦 Installation

No installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

No code examples are provided in the original document, so this section is skipped.

🔧 Technical Details

Details model architecture

This model checkpoint - t5-efficient-small-el2 - is a Small model with the following variations:

• el is 2

It has 47.93 million parameters, requiring ca. 191.72 MB of memory in full precision (fp32) or 95.86 MB in half precision (fp16 or bf16).

Here's a summary of the original T5 model architectures:

If a model checkpoint has no specific el or dl, the number of encoder- and decoder layers equals nl.

Pre-Training

The checkpoint was pretrained on the Colossal, Cleaned version of Common Crawl (C4) for 524288 steps using the span-based masked language modeling (MLM) objective.

Fine-Tuning

⚠️ Important Note

This model is a pretrained checkpoint and needs to be fine-tuned for practical use. It was pretrained in English, so it's only useful for English NLP tasks.

You can follow these examples to fine-tune the model:

PyTorch:

• Summarization
• Question Answering
• Text Classification - Note: You'll need to slightly adapt the training example to work with an encoder-decoder model.

Tensorflow:

• Summarization
• Text Classification - Note: You'll need to slightly adapt the training example to work with an encoder-decoder model.

JAX/Flax:

• Summarization
• Text Classification - Note: You'll need to slightly adapt the training example to work with an encoder-decoder model.

📄 License

The model is released under the apache-2.0 license.