Hindi Tpu Electra

🚀 Hindi Language Model

This is a Hindi language model trained with ELECTRA base size settings, offering high - performance in various NLP tasks.

🚀 Quick Start

You can access the Tokenization and training CoLab here.

✨ Features

Performance in Different Tasks

• Sentiment Analysis: This model outperforms Multilingual BERT on Hindi movie reviews / sentiment analysis (using SimpleTransformers). You can achieve higher accuracy using ktrain + TensorFlow, where you can adjust the learning rate and other hyperparameters: ktrain + TensorFlow Example.
• Question - Answering: It can be used for question - answering on the MLQA dataset: MLQA Question - Answering Example.
• News Classification: A smaller model (Hindi - BERT) performs better on a BBC news classification task.

📦 Installation

No specific installation steps are provided in the original README.

💻 Usage Examples

Base Usage

The usage examples are mainly demonstrated through the Colab notebooks:

• Tokenization and training CoLab
• Hindi movie reviews / sentiment analysis
• ktrain + TensorFlow for higher accuracy
• Question - answering on MLQA dataset

📚 Documentation

Corpus

The corpus consists of two files:

• Hindi CommonCrawl deduped by OSCAR OSCAR Link
• The latest Hindi Wikipedia (Hindi Wikipedia Dumps) + WikiExtractor to txt

Bonus notes:

• Adding English wiki text or parallel corpus could help with cross - lingual tasks and training

Vocabulary

You can access the vocabulary here.

Bonus notes:

• Created with HuggingFace Tokenizers; you can increase vocabulary size and re - train; remember to change ELECTRA vocab_size

Training

Structure your files, with the data - dir named "trainer" here:

trainer
- vocab.txt
- pretrain_tfrecords
-- (all .tfrecord... files)
- models
-- modelname
--- checkpoint
--- graph.pbtxt
--- model.*

Conversion

Use this process to convert an in - progress or completed ELECTRA checkpoint to a Transformers - ready model:

git clone https://github.com/huggingface/transformers
python ./transformers/src/transformers/convert_electra_original_tf_checkpoint_to_pytorch.py
  --tf_checkpoint_path=./models/checkpointdir
  --config_file=config.json
  --pytorch_dump_path=pytorch_model.bin
  --discriminator_or_generator=discriminator
python

from transformers import TFElectraForPreTraining
model = TFElectraForPreTraining.from_pretrained("./dir_with_pytorch", from_pt=True)
model.save_pretrained("tf")

Once you have formed one directory with config.json, pytorch_model.bin, tf_model.h5, special_tokens_map.json, tokenizer_config.json, and vocab.txt on the same level, run:

transformers-cli upload directory