T5 Efficient Tiny

🚀 T5-Efficient-TINY (Deep-Narrow version)

T5-Efficient-TINY is a variant of Google's original T5, adhering to the T5 model architecture. It's a pretrained-only checkpoint, released alongside the paper Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers by Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler.

In essence, the paper suggests that a Deep-Narrow model architecture outperforms other architectures with a similar parameter count in downstream tasks.

Here's a quote from the paper:

We generally recommend a DeepNarrow strategy where the model’s depth is preferentially increased before considering any other forms of uniform scaling across other dimensions. This is largely due to how much depth influences the Pareto-frontier as shown in earlier sections of the paper. Specifically, a tall small (deep and narrow) model is generally more efficient compared to the base model. Likewise, a tall base model might also generally be more efficient compared to a large model. We generally find that, regardless of size, even if absolute performance might increase as we continue to stack layers, the relative gain of Pareto-efficiency diminishes as we increase the layers, converging at 32 to 36 layers. Finally, we note that our notion of efficiency here relates to any one compute dimension, i.e., params, FLOPs or throughput (speed). We report all three key efficiency metrics (number of params, FLOPS and speed) and leave this decision to the practitioner to decide which compute dimension to consider.

More precisely, model depth is defined as the number of sequentially stacked transformer blocks. A sequence of word embeddings is processed sequentially by each transformer block.

🚀 Quick Start

✨ Features

• Based on the T5 model architecture, it offers a Deep-Narrow variant for better downstream performance.
• Pretrained on a large-scale dataset, suitable for English NLP tasks after fine-tuning.

📦 Installation

As this is a model checkpoint in the Hugging Face ecosystem, you can install the necessary dependencies using pip:

pip install transformers

💻 Usage Examples

Since this is a pretrained model that needs fine-tuning, here are some fine-tuning examples in different frameworks:

PyTorch

• Summarization: Summarization Example
• Question Answering: Question Answering Example
• Text Classification: Text Classification Example (Note: Slight adaptation is needed for encoder-decoder models.)

Tensorflow

• Summarization: Summarization Example
• Text Classification: Text Classification Example (Note: Slight adaptation is needed for encoder-decoder models.)

JAX/Flax

• Summarization: Summarization Example
• Text Classification: Text Classification Example (Note: Slight adaptation is needed for encoder-decoder models.)

📚 Documentation

Details model architecture

This model checkpoint - t5-efficient-tiny - is of model type Tiny with no variations. It has 15.58 million parameters and requires ca. 62.32 MB of memory in full precision (fp32) or 31.16 MB of memory in half precision (fp16 or bf16).

Here's a summary of the original T5 model architectures:

Here are the definitions of the abbreviations:

If a model checkpoint has no specific el or dl, both the number of encoder- and decoder layers equal nl.

Pre-Training

The checkpoint was pretrained on the Colossal, Cleaned version of Common Crawl (C4) for 524288 steps using the span-based masked language modeling (MLM) objective.

Fine-Tuning

⚠️ Important Note

This model is a pretrained checkpoint and must be fine-tuned for practical use. It was pretrained in English, so it's only suitable for English NLP tasks.

🔧 Technical Details

The concept of model depth is crucial in this model. It refers to the number of sequentially stacked transformer blocks, which process word embeddings sequentially. The paper suggests that increasing model depth can lead to better Pareto-efficiency, especially in the range of 32 to 36 layers.

📄 License

This model is released under the apache-2.0 license.