Deberta V3 Japanese Xsmall

🚀 DeBERTa V3 Japanese Model

This is a model based on DeBERTa V3 pre-trained on Japanese resources. It addresses the challenges of Japanese language processing and offers high - performance solutions for related tasks.

🚀 Quick Start

from transformers import AutoTokenizer, AutoModelForTokenClassification

model_name = 'globis-university/deberta-v3-japanese-xsmall'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

✨ Features

• Based on the well - known DeBERTa V3 model.
• Specialized for the Japanese language.
• Does not use a morphological analyzer during inference.
• Respects word boundaries to some extent (does not produce tokens spanning multiple words like の都合上 or の判定負けを喫し).

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModelForTokenClassification

model_name = 'globis-university/deberta-v3-japanese-xsmall'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

📚 Documentation

Tokenizer

The tokenizer is trained using the method introduced by Kudo.

Key points include:

• No morphological analyzer needed during inference.
• Tokens do not cross word boundaries (dictionary: unidic-cwj-202302).
• Easy to use with Hugging Face.
• Smaller vocabulary size.

Although the original DeBERTa V3 is characterized by a large vocabulary size, which can result in a significant increase in the number of parameters in the embedding layer (for the microsoft/deberta-v3-base model, the embedding layer accounts for 54% of the total), this model adopts a smaller vocabulary size to address this.

Note that, among the three models: xsmall, base, and large, the first two were trained using the unigram algorithm, while only the large model was trained using the BPE algorithm. The reason for this is simple: while the large model was independently trained to increase its vocabulary size, for some reason, training with the unigram algorithm was not successful. Thus, prioritizing the completion of the model over investigating the cause, we switched to the BPE algorithm.

Data

Training parameters

• Number of devices: 8
• Batch size: 48 x 8
• Learning rate: 3.84e - 4
• Maximum sequence length: 512
• Optimizer: AdamW
• Learning rate scheduler: Linear schedule with warmup
• Training steps: 1,000,000
• Warmup steps: 100,000
• Precision: Mixed (fp16)
• Vocabulary size: 32,000

Evaluation

📄 License

CC BY SA 4.0

Acknowledgement

We used ABCI for computing resources. Thank you.