Electra Small Paper Japanese Generator

🚀 ELECTRA small Japanese generator

This is a pre - trained ELECTRA model for Japanese texts, offering efficient and accurate language processing capabilities.

🚀 Quick Start

This is a ELECTRA model pretrained on texts in the Japanese language. The codes for the pretraining are available at retarfi/language-pretraining.

✨ Features

• Same Architecture: The model architecture is the same as ELECTRA small in the original ELECTRA paper, with 12 layers, 64 dimensions of hidden states, and 1 attention heads.
• Japanese Wikipedia Training: Trained on the Japanese version of Wikipedia, using a dump file as of June 1, 2021. The 2.9GB corpus consists of approximately 20M sentences.
• Tokenization Method: Texts are first tokenized by MeCab with IPA dictionary and then split into sub - words by the WordPiece algorithm, with a vocabulary size of 32768.
• Training Configuration: Trained with the same configuration as ELECTRA small in the original ELECTRA paper, including 128 tokens per instance, 128 instances per batch, and 1M training steps. The generator size is 1/4 of the discriminator size.

📚 Documentation

Model architecture

The model architecture is the same as ELECTRA small in the original ELECTRA paper; 12 layers, 64 dimensions of hidden states, and 1 attention heads.

Training Data

The models are trained on the Japanese version of Wikipedia. The training corpus is generated from the Japanese version of Wikipedia, using Wikipedia dump file as of June 1, 2021. The corpus file is 2.9GB, consisting of approximately 20M sentences.

Tokenization

The texts are first tokenized by MeCab with IPA dictionary and then split into subwords by the WordPiece algorithm. The vocabulary size is 32768.

Training

The models are trained with the same configuration as ELECTRA small in the original ELECTRA paper; 128 tokens per instance, 128 instances per batch, and 1M training steps. The size of the generator is 1/4 of the size of the discriminator.

📄 License

The pretrained models are distributed under the terms of the [Creative Commons Attribution - ShareAlike 4.0](https://creativecommons.org/licenses/by - sa/4.0/).

🔧 Technical Details

Citation

@article{Suzuki-etal-2023-ipm,
  title = {Constructing and analyzing domain-specific language model for financial text mining},
  author = {Masahiro Suzuki and Hiroki Sakaji and Masanori Hirano and Kiyoshi Izumi},
  journal = {Information Processing & Management},
  volume = {60},
  number = {2},
  pages = {103194},
  year = {2023},
  doi = {10.1016/j.ipm.2022.103194}
}

Acknowledgments

This work was supported by JSPS KAKENHI Grant Number JP21K12010.