Hindi Bert

🚀 Releasing Hindi ELECTRA model

This project represents the first attempt to train a Hindi language model using Google Research's ELECTRA. It aims to provide a valuable resource for Hindi language processing tasks.

As of 2022, it is recommended to use Google's MuRIL model, which is trained on English, Hindi, and other major Indian languages in both their script and latinized script: https://huggingface.co/google/muril-base-cased and https://huggingface.co/google/muril-large-cased.

For causal language models, https://huggingface.co/sberbank-ai/mGPT is a good option, though it is a large model.

You can access the Tokenization and training CoLab.

Initially, a modified ELECTRA was used for finetuning, but now SimpleTransformers is adopted.

Check out the Blog post. The project was greatly influenced by https://huggingface.co/blog/how-to-train.

✨ Features

• Comparable results to Multilingual BERT on BBC Hindi news classification and Hindi movie reviews / sentiment analysis.
• Higher accuracy can be achieved using ktrain by adjusting the learning rate.
• A larger model (Hindi-TPU-Electra) shows better performance on Hindi movie reviews / sentiment analysis.

📚 Documentation

🔍 Example Notebooks

• This small model has comparable results to Multilingual BERT on BBC Hindi news classification and on Hindi movie reviews / sentiment analysis (using SimpleTransformers).
• You can get higher accuracy using ktrain by adjusting the learning rate (also: changing model_type in config.json - this is an open issue with ktrain): https://colab.research.google.com/drive/1mSeeSfVSOT7e-dVhPlmSsQRvpn6xC05w?usp=sharing.
• Question-answering on MLQA dataset: https://colab.research.google.com/drive/1i6fidh2tItf_-IDkljMuaIGmEU6HT2Ar#scrollTo=IcFoAHgKCUiQ.
• A larger model (Hindi-TPU-Electra) using ELECTRA base size outperforms both models on Hindi movie reviews / sentiment analysis, but does not perform as well on the BBC news classification task.

📦 Corpus

• Download: https://drive.google.com/drive/folders/1SXzisKq33wuqrwbfp428xeu_hDxXVUUu?usp=sharing.
• The corpus consists of two files:
  • Hindi CommonCrawl deduped by OSCAR https://traces1.inria.fr/oscar/.
  • Latest Hindi Wikipedia (https://dumps.wikimedia.org/hiwiki/) + WikiExtractor to txt.
• Bonus notes: Adding English wiki text or parallel corpus could help with cross-lingual tasks and training.

📖 Vocabulary

• Download: https://drive.google.com/file/d/1-6tXrii3tVxjkbrpSJE9MOG_HhbvP66V/view?usp=sharing.
• Bonus notes: Created with HuggingFace Tokenizers; you can increase the vocabulary size and re-train; remember to change ELECTRA vocab_size.

⚙️ Training

Structure your files, with the data-dir named "trainer" here:

trainer
- vocab.txt
- pretrain_tfrecords
-- (all .tfrecord... files)
- models
-- modelname
--- checkpoint
--- graph.pbtxt
--- model.*

The CoLab notebook provides examples of GPU vs. TPU setup. Refer to configure_pretraining.py.

🔄 Conversion

Use the following process to convert an in-progress or completed ELECTRA checkpoint to a Transformers-ready model:

git clone https://github.com/huggingface/transformers
python ./transformers/src/transformers/convert_electra_original_tf_checkpoint_to_pytorch.py
  --tf_checkpoint_path=./models/checkpointdir
  --config_file=config.json
  --pytorch_dump_path=pytorch_model.bin
  --discriminator_or_generator=discriminator
python

from transformers import TFElectraForPreTraining
model = TFElectraForPreTraining.from_pretrained("./dir_with_pytorch", from_pt=True)
model.save_pretrained("tf")

Once you have a directory with config.json, pytorch_model.bin, tf_model.h5, special_tokens_map.json, tokenizer_config.json, and vocab.txt at the same level, run:

transformers-cli upload directory