Gpt2 Bengali

🚀 Bengali GPT-2

This is a Bengali GPT-2 demo, part of the Huggingface JAX/Flax event. It also features a finetuned model on Bengali song lyrics.

🚀 Quick Start

The Bengali GPT-2 offers multiple ways for usage. For instance, you can directly use the pipeline to generate sentences.

from transformers import pipeline

gpt2_bengali = pipeline('text-generation',model="flax-community/gpt2-bengali", tokenizer='flax-community/gpt2-bengali')

Similarly, for using the finetuned model on Bengali songs, you can use the following code.

from transformers import pipeline

singer = pipeline('text-generation',model="khalidsaifullaah/bengali-lyricist-gpt2", tokenizer='khalidsaifullaah/bengali-lyricist-gpt2')

For other tasks, the model needs to be fine - tuned on custom datasets. Details can be found in the Huggingface documentation

✨ Features

• Bengali Pretraining: The model is pretrained on the Bengali corpus of the mC4 (multilingual C4) dataset, enabling better performance on Bengali language tasks.
• Finetuned Model: There is a finetuned model on Bengali song lyrics, which can be used for generating song - related text.

📦 Installation

The README does not provide specific installation steps, so this section is skipped.

💻 Usage Examples

Basic Usage

from transformers import pipeline

gpt2_bengali = pipeline('text-generation',model="flax-community/gpt2-bengali", tokenizer='flax-community/gpt2-bengali')

Advanced Usage

# For using the finetuned model on Bengali songs
from transformers import pipeline

singer = pipeline('text-generation',model="khalidsaifullaah/bengali-lyricist-gpt2", tokenizer='khalidsaifullaah/bengali-lyricist-gpt2')

📚 Documentation

Model Description

The OpenAI GPT - 2 model was proposed in the Language Models are Unsupervised Multitask Learners paper. The original GPT2 model is a causal (unidirectional) transformer pretrained using language modeling on a very large corpus of ~40 GB of text data. This model has the same configuration but has been pretrained on the Bengali corpus of the mC4 (multilingual C4) dataset. The code for training the model has all been open - sourced here.

Training Details

• Overall Result: Eval loss : 1.45, Eval Perplexity : 3.141
• Data: mC4 - bn
• Train Steps: 250k steps
• Link: flax-community/gpt2-bengali
• Demo: https://huggingface.co/spaces/flax-community/Gpt2-bengali

🔧 Technical Details

The README does not provide detailed technical implementation information, so this section is skipped.

📄 License

This project is licensed under the MIT license.

👥 Contributors

• Khalid Saifullah
• Tasmiah Tahsin Mayeesha
• Ritobrata Ghosh
• Ibrahim Musa
• M Saiful Bari

BibTeX entry and citation info

@misc {flax_community_2023,
author       = { {Flax Community} },
title        = { gpt2-bengali (Revision cb8fff6) },
year         = 2023,
url          = { https://huggingface.co/flax-community/gpt2-bengali },
doi          = { 10.57967/hf/0938 },
publisher    = { Hugging Face }
}