line-distilbert-base-japanese open-source model - Pre-trained on a vast amount of Japanese web pages for precise handling of Japanese tasks

Home

Line Distilbert Base Japanese

Developed by line-corporation

DistilBERT model pre-trained on 131GB of Japanese web text, developed by LINE Corporation

Large Language Model

Transformers

JapaneseOpen Source License:Apache-2.0 #Japanese Text Understanding #Lightweight BERT #JGLUE Benchmark

Downloads 12.92k

Release Time : 3/9/2023

Model Overview

This is a lightweight Japanese BERT model suitable for various natural language processing tasks such as text classification, question answering, and semantic understanding.

Model Features

Lightweight and Efficient

40% fewer parameters than the full BERT model while maintaining over 90% performance

Professional Japanese Processing

Uses MeCab with Unidic dictionary for professional Japanese tokenization

Extensive Pre-training

Pre-trained on 131GB of Japanese web text

Model Capabilities

Japanese Text Understanding

Text Classification

Question Answering Systems

Semantic Similarity Calculation

Masked Language Modeling

Use Cases

Enterprise Applications

Corporate Document Processing

Used for processing and analyzing internal Japanese documents within enterprises

Research and Development

Natural Language Processing Research

Used as a base model for Japanese NLP-related research

🚀 LINE DistilBERT Japanese

This is a DistilBERT model pre - trained on 131 GB of Japanese web text. It is trained by LINE Corporation, with the teacher model being a BERT - base built in - house at LINE.

🚀 Quick Start

How to use

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("line-corporation/line-distilbert-base-japanese", trust_remote_code=True)
model = AutoModel.from_pretrained("line-corporation/line-distilbert-base-japanese")

sentence = "LINE株式会社で[MASK]の研究・開発をしている。"
print(model(**tokenizer(sentence, return_tensors="pt")))

Requirements

fugashi 
sentencepiece
unidic-lite

✨ Features

Pre - trained on 131 GB of Japanese web text.
Teacher model is an in - house BERT - base at LINE.

📚 Documentation

For Japanese

You can find the Japanese version of this README at https://github.com/line/LINE-DistilBERT-Japanese/blob/main/README_ja.md.

🔧 Technical Details

Model architecture

The model architecture is the DitilBERT base model; 6 layers, 768 dimensions of hidden states, 12 attention heads, 66M parameters.

Tokenization

The texts are first tokenized by MeCab with the Unidic dictionary and then split into subwords by the SentencePiece algorithm. The vocabulary size is 32768.

📊 Evaluation

The evaluation by JGLUE is as follows:

model name	#Params	Marc_ja	JNLI	JSTS	JSQuAD	JCommonSenseQA
		acc	acc	Pearson/Spearman	EM/F1	acc
LINE-DistilBERT	68M	95.6	88.9	89.2/85.1	87.3/93.3	76.1
Laboro-DistilBERT	68M	94.7	82.0	87.4/82.7	70.2/87.3	73.2
BandaiNamco-DistilBERT	68M	94.6	81.6	86.8/82.1	80.0/88.0	66.5

📄 License

The pretrained models are distributed under the terms of the Apache License, Version 2.0.

To cite this work

We haven't published any paper on this work. Please cite this GitHub repository:

@article{LINE DistilBERT Japanese,
  title = {LINE DistilBERT Japanese},
  author = {"Koga, Kobayashi and Li, Shengzhe and Nakamachi, Akifumi and Sato, Toshinori"},
  year = {2023},
  howpublished = {\url{http://github.com/line/LINE-DistilBERT-Japanese}}
}

Metadata

Property	Details
Inference	false
Language	Japanese
License	apache - 2.0
Mask Token	"[MASK]"
Widget Text	"LINE株式会社で[MASK]の研究・開発をしている。"

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご