T5 Small Standard Bahasa Cased

🚀 t5-small-standard-bahasa-cased

A pre-trained T5 small standard language model for the Malay language, addressing various language - related tasks.

🚀 Quick Start

You can use this model by installing torch or tensorflow and the Huggingface library transformers. And you can use it directly by initializing it like this:

from transformers import T5Tokenizer, T5Model

model = T5Model.from_pretrained('malay-huggingface/t5-small-bahasa-cased')
tokenizer = T5Tokenizer.from_pretrained('malay-huggingface/t5-small-bahasa-cased')

✨ Features

The t5-small-standard-bahasa-cased model was pretrained on multiple tasks:

1. Language masking task on Malay news, Malay Wikipedia, Malay Academia.edu, Malay parliament, and translated The Pile.
2. News title prediction on Malay news.
3. Next sentence prediction on Malay news, Malay Wikipedia, Malay Academia.edu, Malay parliament, and translated The Pile.
4. Translated QA Natural.
5. Text Similarity task on translated SNLI and translated MNLI.
6. EN - MS translation.
7. MS - EN translation.
8. Abstractive Summarization.
9. Knowledge Graph triples generation.
10. Paraphrase.

The preparation steps can be reproduced at https://github.com/huseinzol05/malaya/tree/master/pretrained-model/t5/prepare

📦 Installation

This model was trained using the Google T5 repository https://github.com/google-research/text-to-text-transfer-transformer, on v3 - 8 TPU. All steps can be reproduced from here, https://github.com/huseinzol05/Malaya/tree/master/pretrained-model/t5

💻 Usage Examples

Basic Usage

from transformers import T5Tokenizer, T5Model

model = T5Model.from_pretrained('malay-huggingface/t5-small-bahasa-cased')
tokenizer = T5Tokenizer.from_pretrained('malay-huggingface/t5-small-bahasa-cased')

Advanced Usage

from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained('malay-huggingface/t5-small-bahasa-cased')
model = T5ForConditionalGeneration.from_pretrained('malay-huggingface/t5-small-bahasa-cased')
input_ids = tokenizer.encode('soalan: siapakah perdana menteri malaysia?', return_tensors = 'pt')
outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))

Output is,

'Mahathir Mohamad'

Supported Prefix

1. soalan: {string}, trained using Natural QA.
2. ringkasan: {string}, for abstractive summarization.
3. tajuk: {string}, for abstractive title.
4. parafrasa: {string}, for abstractive paraphrase.
5. terjemah Inggeris ke Melayu: {string}, for EN - MS translation.
6. terjemah Melayu ke Inggeris: {string}, for MS - EN translation.
7. grafik pengetahuan: {string}, for MS text to EN Knowledge Graph triples format.
8. ayat1: {string1} ayat2: {string2}, semantic similarity.