Gpt2 Small Indonesian

🚀 GPT2-small-indonesian

This is a pre - trained model for the Indonesian language, trained with a causal language modeling (CLM) objective. The CLM objective was first introduced in this paper and first released on this page.

This model was trained using HuggingFace's Flax framework and is part of the JAX/Flax Community Week organized by HuggingFace. All training was conducted on a TPUv3 - 8 VM sponsored by the Google Cloud team.

You can find the demo here.

🚀 Quick Start

✨ Features

• Trained on a combined dataset of Indonesian websites from OSCAR, mc4 and Wikipedia.
• Can be used for text generation and feature extraction in both PyTorch and TensorFlow.

📦 Installation

No specific installation steps are provided in the original README. However, you need to have the transformers library installed to use this model. You can install it using pip install transformers.

💻 Usage Examples

Basic Usage

You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we set a seed for reproducibility:

>>> from transformers import pipeline, set_seed
>>> generator = pipeline('text-generation', model='flax-community/gpt2-small-indonesian')
>>> set_seed(42)
>>> generator("Sewindu sudah kita tak berjumpa,", max_length=30, num_return_sequences=5)

[{'generated_text': 'Sewindu sudah kita tak berjumpa, dua dekade lalu, saya hanya bertemu sekali. Entah mengapa, saya lebih nyaman berbicara dalam bahasa Indonesia, bahasa Indonesia'},
 {'generated_text': 'Sewindu sudah kita tak berjumpa, tapi dalam dua hari ini, kita bisa saja bertemu.”\
“Kau tau, bagaimana dulu kita bertemu?” aku'},
 {'generated_text': 'Sewindu sudah kita tak berjumpa, banyak kisah yang tersimpan. Tak mudah tuk kembali ke pelukan, di mana kini kita berada, sebuah tempat yang jauh'},
 {'generated_text': 'Sewindu sudah kita tak berjumpa, sejak aku lulus kampus di Bandung, aku sempat mencari kabar tentangmu. Ah, masih ada tempat di hatiku,'},
 {'generated_text': 'Sewindu sudah kita tak berjumpa, tapi Tuhan masih saja menyukarkan doa kita masing-masing.\
Tuhan akan memberi lebih dari apa yang kita'}]

Advanced Usage

Here is how to use this model to get the features of a given text in PyTorch:

from transformers import GPT2Tokenizer, GPT2Model
tokenizer = GPT2Tokenizer.from_pretrained('flax-community/gpt2-small-indonesian')
model = GPT2Model.from_pretrained('flax-community/gpt2-small-indonesian')
text = "Ubah dengan teks apa saja."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

and in TensorFlow:

from transformers import GPT2Tokenizer, TFGPT2Model
tokenizer = GPT2Tokenizer.from_pretrained('flax-community/gpt2-small-indonesian')
model = TFGPT2Model.from_pretrained('flax-community/gpt2-small-indonesian')
text = "Ubah dengan teks apa saja."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)

📚 Documentation

Limitations and bias

The training data for this model are from Indonesian websites of OSCAR, mc4 and Wikipedia. These datasets contain a lot of unfiltered internet content, which is far from neutral. Although some filtering has been done on the dataset (see the Training data section), the filtering does not thoroughly mitigate the biased content in the training data. These biases might also affect models fine - tuned using this model.

As the openAI team points out in their model card:

⚠️ Important Note

Because large - scale language models like GPT - 2 do not distinguish fact from fiction, we don’t support use - cases that require the generated text to be true.

⚠️ Important Note

Additionally, language models like GPT - 2 reflect the biases inherent to the systems they were trained on, so we do not recommend that they be deployed into systems that interact with humans unless the deployers first carry out a study of biases relevant to the intended use - case. We found no statistically significant difference in gender, race, and religious bias probes between 774M and 1.5B, implying all versions of GPT - 2 should be approached with similar levels of caution around use cases that are sensitive to biases around human attributes.

We have done a basic bias analysis, which can be found in this notebook. It was performed on Indonesian GPT2 medium, based on the bias analysis for Polish GPT2 with modifications.

Gender bias

We generated 50 texts starting with prompts "She/He works as". After some preprocessing (lowercase and stopwords removal), we obtained texts to generate word clouds of female/male professions. The most salient terms for male professions are: driver, sopir (driver), ojek, tukang, online.

The most salient terms for female professions are: pegawai (employee), konsultan (consultant), asisten (assistant).

Ethnicity bias

We generated 1,200 texts to assess bias across ethnicity and gender vectors. We created prompts with the following scheme:

• Person - We assessed 5 ethnicities: Sunda, Batak, Minahasa, Dayak, Asmat, Neutral (no ethnicity)
• Topic - We used 5 different topics:
  • random act: entered home
  • said: said
  • works as: works as
  • intent: let [person] ...
  • define: is

Sample of generated prompt: "seorang perempuan sunda masuk ke rumah..." (a Sundanese woman enters the house...)

We used a model trained on Indonesian hate speech corpus (dataset 1, dataset 2) to obtain the probability that each generated text contains hate speech. To avoid leakage, we removed the first word identifying the ethnicity and gender from the generated text before running the hate speech detector.

The following chart demonstrates the intensity of hate speech associated with the generated texts with outlier scores removed. Some ethnicities score higher than the neutral baseline.

Religion bias

With the same methodology, we generated 1,400 texts to assess bias across religion and gender vectors. We assessed 6 religions: Islam, Protestan (Protestant), Katolik (Catholic), Buddha (Buddhism), Hindu (Hinduism), and Khonghucu (Confucianism) with Neutral (no religion) as a baseline.

The following chart demonstrates the intensity of hate speech associated with the generated texts with outlier scores removed. Some religions score higher than the neutral baseline.

Training data

The model was trained on a combined dataset of OSCAR, mc4 and Wikipedia for the Indonesian language. We filtered and reduced the mc4 dataset, resulting in a total of 29 GB of data. The mc4 dataset was cleaned using this filtering script, and we only included links cited by the Indonesian Wikipedia.

Training procedure

The model was trained on a TPUv3 - 8 VM provided by the Google Cloud team. The training duration was 4d 14h 50m 47s.

Evaluation results

The model achieves the following results without any fine - tuning (zero - shot):

Tracking

The training process was tracked in TensorBoard and Weights and Biases.

👥 Team members

• Akmal (@Wikidepia)
• alvinwatner (@alvinwatner)
• Cahya Wirawan (@cahya)
• Galuh Sahid (@Galuh)
• Muhammad Agung Hambali (@AyameRushia)
• Muhammad Fhadli (@muhammadfhadli)
• Samsul Rahmadani (@munggok)

🚧 Future work

💡 Usage Tip We would like to pre - train the models further with larger and cleaner datasets and fine - tune it to specific domains if we can get the necessary hardware resources.