Twibert

🚀 TwiBERT

TwiBERT is a pre - trained language model tailored for the Twi language, spoken widely in Ghana, West Africa. It offers efficient language - processing capabilities for the Twi - speaking community.

🚀 Quick Start

You can use TwiBERT by finetuning it on a downstream task. The example code below illustrates how you can use the TwiBERT model on a downstream task:

>>> from transformers import AutoTokenizer, AutoModelForTokenClassification
>>> model = AutoModelForTokenClassification.from_pretrained("sakrah/TwiBERT")
>>> tokenizer = AutoTokenizer.from_pretrained("sakrah/TwiBERT")

✨ Features

TwiBERT is a pre - trained language model specifically designed for the Twi language. This model has 61 million parameters, 6 layers, 6 attention heads, 768 hidden units, and a feed - forward size of 3072. To optimize its performance, TwiBERT was trained using a combination of the Asanti Twi Bible and a dataset sourced through crowdsourcing efforts.

🔧 Technical Details

Model Architecture

TwiBERT has 61 million parameters, 6 layers, 6 attention heads, 768 hidden units, and a feed - forward size of 3072.

Training Data

To optimize its performance, TwiBERT was trained using a combination of the Asanti Twi Bible and a dataset sourced through crowdsourcing efforts.

📄 License

This project is licensed under the MIT license.

⚠️ Important Note

The model was trained on a relatively limited dataset (approximately 5MB), which may hinder its ability to learn intricate contextual embeddings and effectively generalize. Additionally, the dataset's focus on the Bible could potentially introduce a strong religious bias in the model's output.