T5 Efficient Base Ff9000

🚀 T5-Efficient-BASE-FF9000 (Deep-Narrow version)

T5-Efficient-BASE-FF9000 is a variant of Google's original T5, adhering to the T5 model architecture. It's a pretrained-only checkpoint, released alongside the paper Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers by Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler. This model showcases that a Deep-Narrow architecture can enhance downstream performance compared to other models with similar parameter counts.

🚀 Quick Start

This model is a pretrained-only checkpoint. For practical use, it needs to be fine - tuned. It was pretrained in English, so it's suitable for English NLP tasks. You can refer to the following examples for fine - tuning:

PyTorch

• Summarization
• Question Answering
• Text Classification - Note: Adapt the training example for encoder - decoder models.

Tensorflow

• Summarization
• Text Classification - Note: Adapt the training example for encoder - decoder models.

JAX/Flax

• Summarization
• Text Classification - Note: Adapt the training example for encoder - decoder models.

✨ Features

The paper suggests that a Deep-Narrow model architecture is more beneficial for downstream performance than other architectures with similar parameter counts. To quote the paper:

We generally recommend a DeepNarrow strategy where the model’s depth is preferentially increased before considering any other forms of uniform scaling across other dimensions. This is largely due to how much depth influences the Pareto - frontier as shown in earlier sections of the paper. Specifically, a tall small (deep and narrow) model is generally more efficient compared to the base model. Likewise, a tall base model might also generally more efficient compared to a large model. We generally find that, regardless of size, even if absolute performance might increase as we continue to stack layers, the relative gain of Pareto - efficiency diminishes as we increase the layers, converging at 32 to 36 layers. Finally, we note that our notion of efficiency here relates to any one compute dimension, i.e., params, FLOPs or throughput (speed). We report all three key efficiency metrics (number of params, FLOPS and speed) and leave this decision to the practitioner to decide which compute dimension to consider.

📚 Documentation

Details model architecture

This model checkpoint - t5 - efficient - base - ff9000 - is of the Base model type with the following variations:

• ff is 9000

It has 449.42 million parameters, requiring ca. 1797.7 MB of memory in full precision (fp32) or 898.85 MB of memory in half precision (fp16 or bf16).

Model Architecture Table

Abbreviation Definitions

If a model checkpoint has no specific el or dl, both the number of encoder - and decoder layers correspond to nl.

Pre - Training

The checkpoint was pretrained on the Colossal, Cleaned version of Common Crawl (C4) for 524288 steps using the span - based masked language modeling (MLM) objective.

Fine - Tuning

Note: This model is a pretrained checkpoint and must be fine - tuned for practical use. It was pretrained in English, so it's only useful for English NLP tasks.

More information

We highly recommend reading the original paper Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers for a more in - depth understanding of this model checkpoint. As explained in the issue, checkpoints with sh or skv model architecture variations haven't been ported to Transformers due to limited practical use and lack of detailed descriptions. These checkpoints are stored here and might be ported in the future.

📄 License

This model is released under the apache - 2.0 license.