Titulm Mpt 1b V1.0

🚀 TituLM-1B-BN-V1

TituLM-1B-BN-V1 is a large language model specifically designed for generating and understanding Bangla text. Leveraging a decoder-style transformer architecture, it has been comprehensively trained on a dataset containing 4.51 billion Bangla tokens. This model is part of Hishab's iterative training and release series of Bangla LLMs.

🚀 Quick Start

The basic use cases to generate text using this model is simple. Follow the below code to generate text using this model.

📦 Installation

Install the following library before running the code:

pip install transformers
pip install einops
pip install accelerate

💻 Usage Examples

import transformers
from transformers import pipeline

model_name = 'hishab/titulm-1b-bn-v1'

config = transformers.AutoConfig.from_pretrained(model_name, trust_remote_code=True)
config.max_seq_len = 2048

model = transformers.AutoModelForCausalLM.from_pretrained(
  model_name,
  config=config,
  trust_remote_code=True
)

tokenizer = transformers.AutoTokenizer.from_pretrained('hishab/titulm-1b-bn-v1')

pipe = pipeline('text-generation', model=model, tokenizer=tokenizer, device='cuda:0')
output = pipe('আমি বাংলায় গান',
            max_new_tokens=100,
            do_sample=True,
            use_cache=True)

print(output)

✨ Features

• Specifically trained for Bangla text generation and understanding.
• Utilizes a decoder - style transformer architecture.

📚 Documentation

Training

The training process was managed using the robust framework provided by MosaicML's llm - foundry repository. Throughout the training phase, titulm - 1b - bn - v1 underwent a total of 59 iterations, allowing for iterative refinements and optimization.

Notable training configs:

Training evaluation status

• Evaluation CrossEntropy Loss Final loss: 3.11
• Language Perplexity Final Perplexity: 22.562

Datasets

We add Bangla text datasets from several sources including:

• Culturax
• Books
• Bangla Wikipedia
• Banglapedia
• News articles

Our total data size is 58 GB of deduplicated data with 4.51 billion tokens tokenized by our sentencepiece model.

📄 License

This project is licensed under the Apache - 2.0 license.

📚 Citation

@misc{hishab_2024_titulm_1b_bn_v1,
  author = {Hishab Technologies Ltd.},
  title = {TituLM-1B-BN-V1},
  year = {2024},
  publisher = {HuggingFace Models},
  howpublished = {https://huggingface.co/hishab/titulm-1b-bn-v1},
}