Deberta V3 Japanese Large

🚀 [DeBERTa V3 Japanese Large Model]

This is a DeBERTa V3 model pre - trained on Japanese resources. It offers unique features tailored for Japanese language processing, eliminating the need for a morphological analyzer during inference and respecting word boundaries.

🚀 Quick Start

To use this model, you can follow the code example below:

from transformers import AutoTokenizer, AutoModelForTokenClassification

model_name = 'globis-university/deberta-v3-japanese-large'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

✨ Features

• Based on Well - known Architecture: Utilizes the established DeBERTa V3 model.
• Japanese - Specific: Specialized for the Japanese language.
• No Morphological Analyzer: Does not require a morphological analyzer during inference.
• Respect Word Boundaries: To some extent, it respects word boundaries and does not produce tokens spanning multiple words like の都合上 or の判定負けを喫し.

📚 Documentation

Tokenizer

The tokenizer is trained using the method introduced by Kudo. Key points include:

• No Morphological Analyzer Needed: Eliminates the need for a morphological analyzer during inference.
• Respect Word Boundaries: Tokens do not cross word boundaries (dictionary: unidic-cwj-202302).
• Hugging Face Compatibility: Easy to use with Hugging Face.
• Smaller Vocabulary Size: Addresses the issue of excessive embedding layer parameters in the original DeBERTa V3 by adopting a smaller vocabulary size.

Note that among the three models (xsmall, base, and large), the first two were trained using the unigram algorithm, while only the large model was trained using the BPE algorithm. This was due to unsuccessful training with the unigram algorithm when increasing the vocabulary size of the large model.

Data

Training parameters

• Number of devices: 8
• Batch size: 8 x 8
• Learning rate: 6.4e - 5
• Maximum sequence length: 512
• Optimizer: AdamW
• Learning rate scheduler: Linear schedule with warmup
• Training steps: 2,000,000
• Warmup steps: 100,000
• Precision: Mixed (fp16)
• Vocabulary size: 48,000

Evaluation

📄 License

CC BY SA 4.0

🤝 Acknowledgement

We used ABCI for computing resources. Thank you.