Indobert-base-uncased Open-source Model - Optimize Indonesian Language Processing and Excel in Multiple NLP Tasks

Indobert Base Uncased

Developed by indolem

IndoBERT is a BERT model specifically optimized for Indonesian, excelling in multiple Indonesian NLP tasks.

Large Language Model OtherOpen Source License:MIT #Indonesian NLP #Multi-task Benchmark #Pre-trained Model

Downloads 26.35k

Release Time : 3/2/2022

Model Overview

An Indonesian version of the BERT model, used to evaluate the IndoLEM benchmark test set, covering seven tasks in morphosyntax, semantics, and discourse.

Model Features

Indonesian Optimization

Specifically trained for Indonesian, using a corpus of over 220 million Indonesian words.

Excellent Multi-task Performance

Outperforms other models in seven Indonesian NLP tasks, including POS tagging, named entity recognition, and sentiment analysis.

Comparable to English BERT

Achieves a perplexity of 3.97 on the development set, comparable to the English BERT base version.

Model Capabilities

POS Tagging

Named Entity Recognition

Dependency Parsing

Sentiment Analysis

Summarization

Tweet Prediction

Tweet Ranking

Use Cases

Natural Language Processing

Indonesian POS Tagging

Tag parts of speech for words in Indonesian text

96.8% accuracy, outperforming Bi-LSTM and mBERT

Indonesian Named Entity Recognition

Identify named entities in Indonesian text

F1 score of 90.1% on UI dataset and 74.9% on UGM dataset

Indonesian Sentiment Analysis

Analyze sentiment tendencies in Indonesian text

F1 score of 84.13%, better than other comparison models

Social Media Analysis

Next Tweet Prediction

Predict the next tweet an Indonesian user might send

93.7% accuracy

Tweet Ranking

Rank Indonesian tweets by relevance

Spearman correlation coefficient of 0.59

🚀 IndoBERT

IndoBERT is the Indonesian version of the BERT model, trained on a large Indonesian corpus and used to evaluate the IndoLEM benchmark.

🚀 Quick Start

IndoBERT is an Indonesian adaptation of the BERT model. It was trained on a corpus of over 220M words, sourced from three main areas:

Indonesian Wikipedia (74M words)
news articles from Kompas, Tempo (Tala et al., 2003), and Liputan6 (55M words in total)
an Indonesian Web Corpus (Medved and Suchomel, 2017) (90M words).

The model was trained for 2.4M steps (180 epochs), achieving a final perplexity of 3.97 on the development set, comparable to the English BERT-base.

This IndoBERT was employed to evaluate IndoLEM - an Indonesian benchmark encompassing seven tasks for the Indonesian language, covering morpho-syntax, semantics, and discourse.

Task	Metric	Bi-LSTM	mBERT	MalayBERT	IndoBERT
POS Tagging	Acc	95.4	96.8	96.8	96.8
NER UGM	F1	70.9	71.6	73.2	74.9
NER UI	F1	82.2	82.2	87.4	90.1
Dep. Parsing (UD-Indo-GSD)	UAS/LAS	85.25/80.35	86.85/81.78	86.99/81.87	87.12/82.32
Dep. Parsing (UD-Indo-PUD)	UAS/LAS	84.04/79.01	90.58/85.44	88.91/83.56	89.23/83.95
Sentiment Analysis	F1	71.62	76.58	82.02	84.13
Summarization	R1/R2/RL	67.96/61.65/67.24	68.40/61.66/67.67	68.44/61.38/67.71	69.93/62.86/69.21
Next Tweet Prediction	Acc	73.6	92.4	93.1	93.7
Tweet Ordering	Spearman corr.	0.45	0.53	0.51	0.59

The research paper has been published at the 28th COLING 2020. For more information about the benchmarks, please visit https://indolem.github.io.

💻 Usage Examples

Basic Usage

# Load model and tokenizer (tested with transformers==3.5.1)
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("indolem/indobert-base-uncased")
model = AutoModel.from_pretrained("indolem/indobert-base-uncased")

📄 License

This project is licensed under the MIT license.

📚 Citation

If you use our work, please cite:

@inproceedings{koto2020indolem,
  title={IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP},
  author={Fajri Koto and Afshin Rahimi and Jey Han Lau and Timothy Baldwin},
  booktitle={Proceedings of the 28th COLING},
  year={2020}
}

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご