Bert Ascii Medium

🚀 bert-ascii-medium

A medium - sized BERT Language Model pre - trained by predicting the summation of the ASCII code values of the characters in a masked token.

🚀 Quick Start

This is a medium - size BERT Language Model. It is pre - trained with the objective of predicting the summation of the ASCII code values of the characters in a masked token. For more details about the pre - training objective and the pre - training hyperparameters, please refer to [How does the pre - training objective affect what large language models learn about linguistic properties?](https://aclanthology.org/2022.acl - short.16/)

📄 License

CC BY 4.0

📚 Documentation

If you use this model, please cite the following paper:

@inproceedings{alajrami2022does,
  title={How does the pre - training objective affect what large language models learn about linguistic properties?},
  author={Alajrami, Ahmed and Aletras, Nikolaos},
  booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  pages={131--147},
  year={2022}
}