Mirror Roberta Base Sentence Drophead

🚀 cambridgeltl/mirror-roberta-base-sentence-drophead

An unsupervised sentence encoder leveraging drophead for feature space augmentation, trained on unlabelled raw sentences with roberta-base as the base model.

🚀 Quick Start

An unsupervised sentence encoder proposed by Liu et al. (2021), using drophead instead of dropout as feature space augmentation. The model is trained with unlabelled raw sentences, using roberta-base as the base model. Please use [CLS] (before pooler) as the representation of the input.

⚠️ Important Note

Note the model does not replicate the exact numbers in the paper since the reported numbers in the paper are average of three runs.

📄 License

Citation

@inproceedings{
	liu2021fast,
  title={Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders},
  author={Liu, Fangyu and Vuli{\'c}, Ivan and Korhonen, Anna and Collier, Nigel},
  booktitle={EMNLP 2021},
  year={2021}
}