Modern Bert Multilingual

🚀 ModernBertMultilingual

ModernBertMultilingual is a multilingual model trained from scratch, supporting four languages including Chinese, English, Japanese, and Korean. It performs well in handling mixed East Asian language text tasks.

✨ Features

• ModernBertMultilingual is a multilingual model trained from scratch.
• It uses the ModernBERT-base architecture.
• It supports four languages and their variants, including Chinese (Simplified, Traditional), English, Japanese, and Korean.
• It performs well on mixed East Asian language text tasks.

🔧 Technical Details

• It uses a slightly adjusted vocabulary from the Qwen2.5 series to support multilingualism.
• It was trained for approximately 100 hours on L40*7 devices, with a training volume of about 60B tokens.
• Key training parameters:
  • Batch Size: 1792
  • Learning Rate: 5e-04
  • Maximum Sequence Length: 512
  • Optimizer: adamw_torch
  • LR Scheduler: warmup_stable_decay
  • Train Precision: bf16 mix
• For other technical specifications, please refer to the original release information and paper of ModernBERT-base.

📚 Documentation

Released Versions

• It provides 3 different weight versions:
  • base - Fully trained with general corpus, suitable for various text domains.
  • nodecay - Checkpoint before the annealing stage. You can fine - tune it with domain - specific data to better adapt to target domains.
  • keyword_gacha_multilingual - Fine - tuned version using ACGN (e.g., light novels, game text, manga text, etc.) type text.

Other Information

• Training script: Github

📄 License

This project is licensed under the apache-2.0 license.