Pko T5 Base

🚀 pko-t5-base

pko-t5-base is a T5 v1.1 model trained on Korean-specific data, aiming to provide excellent performance in Korean language processing.

Source Code

pko-t5 is a T5 v1.1 model trained with Korean-only data.

Instead of SentencePiece, it uses Byte Pair Encoding (BBPE) without Out-Of-Vocabulary (OOV) issues to tokenize Korean. The model was trained using only unsupervised learning on Korean data (such as NamuWiki, Wikipedia, and the National Institute of Korean Language corpus) with T5's span corruption task.

When using pko-t5, it is recommended to fine-tune it for your target task.

🚀 Quick Start

You can access the pko-t5 model through the transformers API. When using the tokenizer, please use T5TokenizerFast instead of T5Tokenizer. You can directly use T5ForConditionalGeneration as the model.

💻 Usage Examples

Basic Usage

from transformers import T5TokenizerFast, T5ForConditionalGeneration

tokenizer = T5TokenizerFast.from_pretrained('paust/pko-t5-base')
model = T5ForConditionalGeneration.from_pretrained('paust/pko-t5-base')

input_ids = tokenizer(["qa question: 당신의 이름은 무엇인가요?"]).input_ids
labels = tokenizer(["T5 입니다."]).input_ids
outputs = model(input_ids=input_ids, labels=labels)

print(f"loss={outputs.loss} logits={outputs.logits}")

📊 Klue Evaluation (dev)

• FT: Single-task fine-tuning / MT: Multi-task fine-tuning
• Baseline: The SOTA scores on the dev set introduced in the KLUE paper.

📄 License

pko-t5, developed by PAUST, is released under the MIT license.