albert-base-japanese-v1 Open-Source Japanese Model - Conveniently Process Japanese Texts with a Japanese Word Segmenter

Albert Base Japanese V1 With Japanese Tokenizer

Developed by ken11

This is a Japanese-pretrained ALBERT model that uses BertJapaneseTokenizer as its tokenizer, making Japanese text processing more convenient.

Large Language Model

Transformers

JapaneseOpen Source License:MIT #Japanese Masked Language Modeling #ALBERT Architecture #Wikipedia Pretraining

Downloads 44

Release Time : 4/20/2022

Model Overview

This model is a Japanese-pretrained ALBERT-based model primarily designed for masked language modeling tasks in Japanese. It can be fine-tuned for various natural language processing tasks.

Model Features

Japanese-optimized Tokenizer

Uses BertJapaneseTokenizer, which provides more efficient and convenient processing of Japanese text compared to the original model.

Lightweight Architecture

Based on the ALBERT architecture, parameter sharing makes the model more lightweight and efficient.

Easy Fine-tuning

The pretrained model is designed for fine-tuning on various downstream tasks, offering strong adaptability.

Model Capabilities

Japanese Text Understanding

Masked Language Prediction

Text Feature Extraction

Use Cases

Text Completion

Japanese Proverb Completion

Completes missing parts in Japanese proverbs, e.g., '明日は明日の[MASK]が吹く'

Can predict suitable completion words like '風'.

Natural Language Processing

Downstream Task Fine-tuning

Can be used as a base model for fine-tuning NLP tasks like text classification and named entity recognition.

🚀 ALBERT Base Japanese V1 with Japanese

This is a pre-trained ALBERT model for the Japanese language. It uses the BertJapaneseTokenizer class for tokenization, making the tokenization process easier compared to albert-base-japanese-v1.

🚀 Quick Start

This model is a pre-trained model, which is generally expected to be fine-tuned for various tasks.

✨ Features

Utilizes the BertJapaneseTokenizer class for easier tokenization.
Suitable for fine-tuning on various Japanese language tasks.

💻 Usage Examples

Basic Usage

This model is a PreTrained model. It is basically assumed to be fine-tuned for various tasks.

Advanced Usage

for PyTorch

from transformers import (
    AutoModelForMaskedLM, AutoTokenizer
)

tokenizer = AutoTokenizer.from_pretrained("ken11/albert-base-japanese-v1-with-japanese-tokenizer")
model = AutoModelForMaskedLM.from_pretrained("ken11/albert-base-japanese-v1-with-japanese-tokenizer")

text = "明日は明日の[MASK]が吹く"
tokens = tokenizer(text, return_tensors="pt")
mask_index = tokens["input_ids"][0].tolist().index(tokenizer.mask_token_id)
predict = model(**tokens)[0]
_, result = predict[0, mask_index].topk(5)

print(tokenizer.convert_ids_to_tokens(result.tolist()))

for TensorFlow

from transformers import (
    TFAutoModelForMaskedLM, AutoTokenizer
)
import tensorflow as tf

tokenizer = AutoTokenizer.from_pretrained("ken11/albert-base-japanese-v1-with-japanese-tokenizer")
model = TFAutoModelForMaskedLM.from_pretrained("ken11/albert-base-japanese-v1-with-japanese-tokenizer")

text = "明日は明日の[MASK]が吹く"
tokens = tokenizer(text, return_tensors="tf")
mask_index = tokens["input_ids"][0].numpy().tolist().index(tokenizer.mask_token_id)
predict = model(**tokens)[0]
result = tf.math.top_k(predict[0, mask_index], k=5)

print(tokenizer.convert_ids_to_tokens(result.indices.numpy()))

📚 Documentation

Training Data

The following data was used for training:

Full text of Japanese Wikipedia

Tokenizer

The tokenizer uses the BertJapaneseTokenizer class. The training data for the tokenizer is the same as that for the model.

📄 License

The MIT license

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご