Muppet Roberta Base

🚀 Muppet: Massive Multi-task Representations with Pre-Finetuning (RoBERTa base model)

This project presents a Massive Multi-task Pre-finetuned version of the RoBERTa base model. It enhances performance across a wide range of GLUE and QA tasks, especially showing significant gains in smaller datasets.

🚀 Quick Start

The model is mainly designed to be fine - tuned on downstream tasks. You can explore fine - tuned versions on the model hub.

✨ Features

• Bidirectional Representation: Learns a bidirectional representation of sentences through Masked Language Modeling (MLM), different from traditional RNNs and autoregressive models.
• Improved Performance: Outperforms the Roberta - base model in various GLUE and QA tasks, as shown in the evaluation results.

📚 Documentation

Model Description

RoBERTa is a self - supervised transformers model pretrained on a large English corpus. It uses the Masked Language Modeling (MLM) objective. In MLM, the model randomly masks 15% of the words in the input sentence, processes the masked sentence, and predicts the masked words. This enables the model to learn a bidirectional representation of the sentence, which can be used for downstream tasks.

Intended Uses & Limitations

• Intended Uses: Primarily for fine - tuning on tasks that use the whole sentence (potentially masked) for decision - making, such as sequence classification, token classification, or question answering.
• Limitations: Not suitable for text generation tasks. For such tasks, models like GPT2 should be considered.

Evaluation Results

When fine - tuned on downstream tasks, the model achieves the following results:

Glue test results:

BibTeX entry and citation info

@article{DBLP:journals/corr/abs-2101-11038,
  author    = {Armen Aghajanyan and
               Anchit Gupta and
               Akshat Shrivastava and
               Xilun Chen and
               Luke Zettlemoyer and
               Sonal Gupta},
  title     = {Muppet: Massive Multi-task Representations with Pre-Finetuning},
  journal   = {CoRR},
  volume    = {abs/2101.11038},
  year      = {2021},
  url       = {https://arxiv.org/abs/2101.11038},
  archivePrefix = {arXiv},
  eprint    = {2101.11038},
  timestamp = {Sun, 31 Jan 2021 17:23:50 +0100},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2101-11038.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

📄 License

This project is licensed under the MIT license.

⚠️ Important Note

This checkpoint does not contain the classification/MRC heads used during pre - finetuning due to compatibility issues. So, you might get slightly lower performance than that reported in the paper on some datasets.

💡 Usage Tip

You can use the raw model for masked language modeling, but it's mostly intended to be fine - tuned on a downstream task. Look for fine - tuned versions on the model hub according to your interests.