K 12BERT

🚀 K-12BERT model

K-12BERT is a model designed for the K-12 education domain. It addresses the lack of a suitable BERT - like model in this field by performing continued pretraining on the K-12Corpus. This model can contribute to advancing AI in education and provide valuable resources for the next generation.

🚀 Quick Start

✨ Features

• Domain - specific: K-12BERT is trained on a custom - curated dataset from both open and proprietary education resources, making it well - suited for the K-12 education domain.
• Resource - efficient: Trained using continued pretraining with an MLM objective, it saves computational resources and leverages existing BERT knowledge.
• Vocabulary preservation: The original BERT vocabulary is preserved to evaluate performance under specific conditions.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

Here is how to use this model to get the features of a given text in PyTorch:

from transformers import BertTokenizer, BertModel, AutoTokenizer, AutoModelForMaskedLM
tokenizer = BertTokenizer.from_pretrained('vasugoel/K-12BERT') # AutoTokenizer.from_pretrained('vasugoel/K-12BERT')
model = BertModel.from_pretrained("vasugoel/K-12BERT") # AutoModelForMaskedLM.from_pretrained('vasugoel/K-12BERT')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

📚 Documentation

Intended uses

We hope that the community, especially researchers and professionals in the education domain, can utilize this model to advance AI in education. With numerous applications for online education platforms, we aim to contribute to the development of education resources for the upcoming generation.

🔧 Technical Details

The model was trained using an MLM objective and in a continued pretraining fashion because of the lack of resources to train it from scratch. This approach not only saved a lot of computational resources but also allowed us to utilize the existing knowledge of BERT. We also preserved the original vocabulary of BERT to evaluate the performance under those conditions.

📄 License

This project is licensed under the Apache 2.0 license.

BibTeX entry and citation info

@misc{https://doi.org/10.48550/arxiv.2205.12335,
  doi = {10.48550/ARXIV.2205.12335},
  
  url = {https://arxiv.org/abs/2205.12335},
  
  author = {Goel, Vasu and Sahnan, Dhruv and V, Venktesh and Sharma, Gaurav and Dwivedi, Deep and Mohania, Mukesh},
  
  keywords = {Computation and Language (cs.CL), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
  
  title = {K-12BERT: BERT for K-12 education},
  
  publisher = {arXiv},
  
  year = {2022},
  
  copyright = {arXiv.org perpetual, non-exclusive license}
}

Additional Information