Kochatbart

🚀 😎 KoChatBART

BART (Bidirectional and Auto-Regressive Transformers) is trained in the form of an autoencoder that adds noise to a part of the input text and restores it to the original text. Korean Chat BART (hereinafter referred to as KoChatBART) is a Korean encoder-decoder language model trained on over 10GB of Korean conversation text using the Text Infilling noise function used in the paper. We are releasing KoChatBART-base, which is robust for dialogue generation.

🚀 Quick Start

from transformers import AutoTokenizer, BartForConditionalGeneration
  
tokenizer = AutoTokenizer.from_pretrained("BM-K/KoChatBART")
model = BartForConditionalGeneration.from_pretrained("BM-K/KoChatBART")

inputs = tokenizer("안녕 세상아!", return_tensors="pt")
outputs = model(**inputs)

📚 Documentation

Pre - processing of Pretrained Data

The datasets used are:

• Topic - based Text Daily Conversation Data
• Small Business Customer Order Q&A Text
• Korean SNS
• Civil Complaint Work Automation AI Language Data

To train KoChatBART, we pre - processed and combined Korean conversation datasets to create a large Korean conversation corpus:

1. To reduce data duplication, when repeated expressions like 'ㅋㅋㅋㅋㅋㅋ' appear more than twice, we changed them to 'ㅋㅋ'.
2. Since very short data can interfere with training, we selected only data with a total token length greater than 3 according to the KoBART tokenizer.
3. We removed anonymized data.

Model

Measuring Dialogue Generation Performance

Each model was fine - tuned based on the following code (Dialogue Generator). To measure the dialogue generation performance, after restoring the generated responses that were tokenized during inference, we used the BPE tokenizer to measure the overlap and distinctness between the actual responses and the generated responses.

⚠️ Important Note

Since the model is generally pre - trained with short dialogue data, it performs poorly on tasks that require long - sentence processing (such as summarization).

Experiment Results

• Emotional Dialogue Data

• Small Business Dialogue Data

👥 Contributors

📖 Reference

• KoBART