Bert Fc Medium

🚀 bert-fc-medium

A medium-size BERT Language Model with a first character prediction pre-training objective. This model offers insights into how pre-training objectives influence what large language models learn about linguistic properties.

🚀 Quick Start

This is a medium-sized BERT language model. Its pre-training objective focuses on first character prediction. For more details about the pre-training objective and hyperparameters, please refer to How does the pre-training objective affect what large language models learn about linguistic properties?

📄 License

CC BY 4.0

📚 Documentation

Citation

If you use this model, please cite the following paper:

@inproceedings{alajrami2022does,
  title={How does the pre-training objective affect what large language models learn about linguistic properties?},
  author={Alajrami, Ahmed and Aletras, Nikolaos},
  booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  pages={131--147},
  year={2022}
}