🚀 T5-Efficient-BASE (Deep-Narrow version)
T5-Efficient-BASE is a variant of Google's original T5, following the T5 model architecture. It's a pretrained-only checkpoint, released with the paper Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers by Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler. This model shows that a Deep-Narrow architecture can achieve better downstream performance with a similar parameter count.
🚀 Quick Start
This is a pretrained checkpoint. For practical use, it needs to be fine - tuned. It was pretrained in English, so it's suitable for English NLP tasks. You can refer to the following examples for fine - tuning:
PyTorch
Tensorflow
JAX/Flax
✨ Features
The paper suggests that a Deep-Narrow model architecture is more favorable for downstream performance compared to other architectures with a similar parameter count. To quote the paper:
We generally recommend a DeepNarrow strategy where the model’s depth is preferentially increased before considering any other forms of uniform scaling across other dimensions. This is largely due to how much depth influences the Pareto - frontier as shown in earlier sections of the paper. Specifically, a tall small (deep and narrow) model is generally more efficient compared to the base model. Likewise, a tall base model might also generally more efficient compared to a large model. We generally find that, regardless of size, even if absolute performance might increase as we continue to stack layers, the relative gain of Pareto - efficiency diminishes as we increase the layers, converging at 32 to 36 layers. Finally, we note that our notion of efficiency here relates to any one compute dimension, i.e., params, FLOPs or throughput (speed). We report all three key efficiency metrics (number of params, FLOPS and speed) and leave this decision to the practitioner to decide which compute dimension to consider.
📚 Documentation
Details model architecture
The t5 - efficient - base model checkpoint is of type Base with no variations. It has 222.93 million parameters, requiring ca. 891.73 MB of memory in full precision (fp32) or 445.86 MB in half precision (fp16 or bf16).
A summary of the original T5 model architectures:
Model |
nl (el/dl) |
ff |
dm |
kv |
nh |
#Params |
Tiny |
4/4 |
1024 |
256 |
32 |
4 |
16M |
Mini |
4/4 |
1536 |
384 |
32 |
8 |
31M |
Small |
6/6 |
2048 |
512 |
32 |
8 |
60M |
Base |
12/12 |
3072 |
768 |
64 |
12 |
220M |
Large |
24/24 |
4096 |
1024 |
64 |
16 |
738M |
Xl |
24/24 |
16384 |
1024 |
128 |
32 |
3B |
XXl |
24/24 |
65536 |
1024 |
128 |
128 |
11B |
Abbreviations used:
Property |
Details |
nl |
Number of transformer blocks (depth) |
dm |
Dimension of embedding vector (output vector of transformers block) |
kv |
Dimension of key/value projection matrix |
nh |
Number of attention heads |
ff |
Dimension of intermediate vector within transformer block (size of feed - forward projection matrix) |
el |
Number of transformer blocks in the encoder (encoder depth) |
dl |
Number of transformer blocks in the decoder (decoder depth) |
sh |
Signifies that attention heads are shared |
skv |
Signifies that key - values projection matrices are tied |
If a model checkpoint has no specific el or dl, both the number of encoder - and decoder layers correspond to nl.
Pre - Training
The checkpoint was pretrained on the Colossal, Cleaned version of Common Crawl (C4) for 524288 steps using the span - based masked language modeling (MLM) objective.
Fine - Tuning
⚠️ Important Note
This model is a pretrained checkpoint and must be fine - tuned for practical use. It was pretrained in English, so it's only useful for English NLP tasks.
🔧 Technical Details
Model depth is defined as the number of transformer blocks stacked sequentially. A sequence of word embeddings is processed sequentially by each transformer block.
📄 License
This project is licensed under the Apache 2.0 license.