T5 Efficient Large Nh32

🚀 T5-Efficient-LARGE-NH32 (Deep-Narrow version)

T5-Efficient-LARGE-NH32 is a variation of Google's original T5 that follows the T5 model architecture. It is a pretrained-only checkpoint released with the paper Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers by Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler. This model demonstrates that a Deep-Narrow architecture can offer better downstream performance compared to other architectures with a similar parameter count.

🚀 Quick Start

This is a pretrained-only model. To use it for practical NLP tasks, you need to fine-tune it. You can follow the fine-tuning examples provided in different frameworks below.

✨ Features

• Deep-Narrow Architecture: The paper suggests that increasing model depth before other forms of scaling can lead to better Pareto-efficiency in terms of params, FLOPs, or throughput.
• Pretrained on C4: The model was pretrained on the Colossal, Cleaned version of Common Crawl (C4) using span-based masked language modeling.

📦 Installation

No specific installation steps are provided in the original README.

💻 Usage Examples

Since this is a pretrained-only model, you need to fine-tune it for specific tasks. Here are some fine-tuning examples in different frameworks:

PyTorch

• Summarization
• Question Answering
• Text Classification - Note: You will have to slightly adapt the training example here to make it work with an encoder-decoder model.

Tensorflow

• Summarization
• Text Classification - Note: You will have to slightly adapt the training example here to make it work with an encoder-decoder model.

JAX/Flax

• Summarization
• Text Classification - Note: You will have to slightly adapt the training example here to make it work with an encoder-decoder model.

📚 Documentation

Details model architecture

This model checkpoint - t5-efficient-large-nh32 - is of model type Large with the following variations:

• nh is 32

It has 1039.72 million parameters and thus requires ca. 4158.86 MB of memory in full precision (fp32) or 2079.43 MB of memory in half precision (fp16 or bf16).

A summary of the original T5 model architectures can be seen here:

The following abbreviations are used:

If a model checkpoint has no specific, el or dl than both the number of encoder- and decoder layers correspond to nl.

Pre-Training

The checkpoint was pretrained on the Colossal, Cleaned version of Common Crawl (C4) for 524288 steps using the span-based masked language modeling (MLM) objective.

Fine-Tuning

Note: This model is a pretrained checkpoint and has to be fine-tuned for practical usage. The checkpoint was pretrained in English and is therefore only useful for English NLP tasks.

More information

We strongly recommend the reader to go carefully through the original paper Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers to get a more nuanced understanding of this model checkpoint. As explained in the following issue, checkpoints including the sh or skv model architecture variations have not been ported to Transformers as they are probably of limited practical usage and are lacking a more detailed description. Those checkpoints are kept here as they might be ported potentially in the future.

🔧 Technical Details

To be more precise, model depth is defined as the number of transformer blocks that are stacked sequentially. A sequence of word embeddings is therefore processed sequentially by each transformer block.

The paper indicates that a Deep-Narrow model architecture is favorable for downstream performance compared to other model architectures of similar parameter count. To quote the paper:

We generally recommend a DeepNarrow strategy where the model’s depth is preferentially increased before considering any other forms of uniform scaling across other dimensions. This is largely due to how much depth influences the Pareto-frontier as shown in earlier sections of the paper. Specifically, a tall small (deep and narrow) model is generally more efficient compared to the base model. Likewise, a tall base model might also generally more efficient compared to a large model. We generally find that, regardless of size, even if absolute performance might increase as we continue to stack layers, the relative gain of Pareto-efficiency diminishes as we increase the layers, converging at 32 to 36 layers. Finally, we note that our notion of efficiency here relates to any one compute dimension, i.e., params, FLOPs or throughput (speed). We report all three key efficiency metrics (number of params, FLOPS and speed) and leave this decision to the practitioner to decide which compute dimension to consider.

📄 License

This model is released under the apache-2.0 license.