Bert Small Japanese Fin

🚀 BERT small Japanese finance

This is a pre - trained BERT model on Japanese texts, which can be used for financial text mining and related tasks.

🚀 Quick Start

This is a BERT model pretrained on texts in the Japanese language. The codes for the pretraining are available at retarfi/language-pretraining.

✨ Features

• The model architecture is based on BERT small, suitable for Japanese text processing.
• Trained on both Wikipedia and financial corpora, enhancing its performance in the financial domain.

📚 Documentation

Model architecture

The model architecture is the same as BERT small in the original ELECTRA paper; 12 layers, 256 dimensions of hidden states, and 4 attention heads.

Training Data

Tokenization

The texts are first tokenized by MeCab with IPA dictionary and then split into subwords by the WordPiece algorithm. The vocabulary size is 32768.

Training

The models are trained with the same configuration as BERT small in the original ELECTRA paper; 128 tokens per instance, 128 instances per batch, and 1.45M training steps.

Citation

@article{Suzuki-etal-2023-ipm,
  title = {Constructing and analyzing domain-specific language model for financial text mining}
  author = {Masahiro Suzuki and Hiroki Sakaji and Masanori Hirano and Kiyoshi Izumi},
  journal = {Information Processing & Management},
  volume = {60},
  number = {2},
  pages = {103194},
  year = {2023},
  doi = {10.1016/j.ipm.2022.103194}
}

📄 License

The pretrained models are distributed under the terms of the Creative Commons Attribution - ShareAlike 4.0.

🔧 Technical Details

The model is built on the BERT small architecture, which has been proven effective in natural language processing tasks. By training on a large - scale Japanese Wikipedia corpus and a financial corpus, the model can better understand the semantics and context of Japanese financial texts. The tokenization process using MeCab and WordPiece algorithms helps to handle the complex Japanese language structure.

Acknowledgments

This work was supported by JSPS KAKENHI Grant Number JP21K12010.