Bert Base Uncased Sst2 From Bert Large Uncased Sst2

🚀 BERT-base-uncased Fine-tuned on SST-2 Dataset

This project presents a fine-tuned bert-base-uncased model on the SST-2 dataset, leveraging knowledge distillation techniques. It uses a fine-tuned bert-large-uncased as a teacher model and tools like torchdistill and Google Colab for the process.

🚀 Quick Start

bert-base-uncased is fine-tuned on the SST-2 dataset. The fine-tuning process employs the fine-tuned bert-large-uncased as a teacher model. Knowledge distillation is carried out using torchdistill and Google Colab.

The training configuration, including hyperparameters, can be found here.

The prediction files were submitted to the GLUE leaderboard, achieving an overall GLUE score of 78.9.

📚 Documentation

Yoshitomo Matsubara presented the paper "torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP" at EMNLP 2023 Workshop for Natural Language Processing Open Source Software (NLP-OSS).

• [Paper]
• [OpenReview]
• [Preprint]

@inproceedings{matsubara2023torchdistill,
  title={{torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP}},
  author={Matsubara, Yoshitomo},
  booktitle={Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)},
  publisher={Empirical Methods in Natural Language Processing},
  pages={153--164},
  year={2023}
}

📄 License

This project is licensed under the apache-2.0 license.

📦 Information Table