Mambarim 110m

🚀 Mambarim-110M

The first Portuguese language model based on a state - space model architecture (Mamba), not a transformer.

🚀 Quick Start

You need to install transformers from main until transformers=4.39.0 is released.

pip install git+https://github.com/huggingface/transformers@main

We also recommend you to install both causal_conv_1d and mamba-ssm using:

pip install causal-conv1d>=1.2.0
pip install mamba-ssm

✨ Features

Mambarim-110M is the first Portuguese language model based on a state - space model architecture (Mamba), not a transformer.

📚 Documentation

Details

• Architecture: a Mamba model pre - trained via causal language modeling
• Size: 119,930,880 parameters
• Context length: 2048 tokens
• Dataset: Pt - Corpus Instruct (6.2B tokens)
• Language: Portuguese
• Number of steps: 758,423

This repository has the source code used to train this model.

Intended Uses

WIP

Out - of - scope Use

WIP

💻 Usage Examples

Basic Usage

You can use the classic generate API:

>>> from transformers import MambaConfig, MambaForCausalLM, AutoTokenizer
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("dominguesm/mambarim-110m")
>>> model = MambaForCausalLM.from_pretrained("dominguesm/mambarim-110m")
>>> input_ids = tokenizer("O Natal é uma", return_tensors="pt")["input_ids"]
>>> out = model.generate(
    input_ids,
    repetition_penalty=1.2,
    temperature=0.8,
    top_k=50,
    top_p=0.85,
    do_sample=True,
    max_new_tokens=10
)
>>> print(tokenizer.batch_decode(out))
["<s> O Natal é uma data em que as pessoas passam horas de lazer e"]

📊 Benchmarks

Evaluations on Brazilian Portuguese benchmarks were performed using a Portuguese implementation of the EleutherAI LM Evaluation Harness (created by Eduardo Garcia).

Detailed results can be found here

📄 License

The model is released under the cc - by - 4.0 license.