🚀 Sabiá-7B: A Portuguese Language Model
Sabiá-7B is a Portuguese language model developed by Maritaca AI. It offers text generation capabilities based on a specific architecture and training data, with certain limitations and performance metrics.
🚀 Quick Start
Sabiá-7B is designed to accept text input and generate text output. Here are some key aspects to get you started:
Input and Output
- Input: The model accepts only text input.
- Output: The model generates text only.
Model Architecture and Tokenizer
- Model Architecture: Sabiá-7B is an auto-regressive language model that uses the same architecture as LLaMA-1-7B.
- Tokenizer: It uses the same tokenizer as LLaMA-1-7B.
Maximum Sequence Length
The maximum sequence length supported by the model is 2048 tokens.
Pretraining Data
- The model was pretrained on 7 billion tokens from the Portuguese subset of ClueWeb22, starting with the weights of LLaMA-1-7B.
- It was further trained for an additional 10 billion tokens, approximately 1.4 epochs of the training dataset.
- The pretraining data has a cutoff of mid - 2022.
License
The licensing is the same as LLaMA-1's, restricting the model's use to research purposes only.
Paper
For more details, please refer to our paper: Sabiá: Portuguese Large Language Models
✨ Features
- Auto - Regressive Architecture: Based on the same architecture as LLaMA-1-7B, providing a solid foundation for text generation.
- Portuguese Focus: Pretrained on Portuguese data, making it suitable for Portuguese language tasks.
💻 Usage Examples
Basic Usage
Given that Sabiá-7B was trained solely on a language modeling objective without fine - tuning for instruction following, it is recommended for few - shot tasks rather than zero - shot tasks, like in the example below.
import torch
from transformers import LlamaTokenizer, LlamaForCausalLM
tokenizer = LlamaTokenizer.from_pretrained("maritaca-ai/sabia-7b")
model = LlamaForCausalLM.from_pretrained(
"maritaca-ai/sabia-7b",
device_map="auto",
low_cpu_mem_usage=True,
torch_dtype=torch.bfloat16
)
prompt = """Classifique a resenha de filme como "positiva" ou "negativa".
Resenha: Gostei muito do filme, é o melhor do ano!
Classe: positiva
Resenha: O filme deixa muito a desejar.
Classe: negativa
Resenha: Apesar de longo, valeu o ingresso.
Classe:"""
input_ids = tokenizer(prompt, return_tensors="pt")
output = model.generate(
input_ids["input_ids"].to("cuda"),
max_length=1024,
eos_token_id=tokenizer.encode("\n"))
output = output[0][len(input_ids["input_ids"][0]):]
print(tokenizer.decode(output, skip_special_tokens=True))
Advanced Usage
If your GPU does not have enough RAM, try using int8 precision. However, expect some degradation in the model output quality when compared to fp16 or bf16.
model = LlamaForCausalLM.from_pretrained(
"maritaca-ai/sabia-7b",
device_map="auto",
low_cpu_mem_usage=True,
load_in_8bit=True,
)
📚 Documentation
Results in Portuguese
Below we show the results on the Poeta benchmark, which consists of 14 Portuguese datasets. For more information on the Normalized Preferred Metric (NPM), please refer to our paper.
Model |
NPM |
LLaMA-1-7B |
33.0 |
LLaMA-2-7B |
43.7 |
Sabiá-7B |
48.5 |
Results in English
Below we show the average results on 6 English datasets: PIQA, HellaSwag, WinoGrande, ARC - e, ARC - c, and OpenBookQA.
Model |
NPM |
LLaMA-1-7B |
50.1 |
Sabiá-7B |
49.0 |
Open Portuguese LLM Leaderboard Evaluation Results
Detailed results can be found here
Metric |
Value |
Average |
47.09 |
ENEM Challenge (No Images) |
55.07 |
BLUEX (No Images) |
47.71 |
OAB Exams |
41.41 |
Assin2 RTE |
46.68 |
Assin2 STS |
1.89 |
FaQuAD NLI |
58.34 |
HateBR Binary |
61.93 |
PT Hate Speech Binary |
64.13 |
tweetSentBR |
46.64 |
📄 License
The licensing is the same as LLaMA-1's, restricting the model's use to research purposes only.
📚 Citation
Please use the following bibtex to cite our paper:
@InProceedings{10.1007/978-3-031-45392-2_15,
author="Pires, Ramon
and Abonizio, Hugo
and Almeida, Thales Sales
and Nogueira, Rodrigo",
editor="Naldi, Murilo C.
and Bianchi, Reinaldo A. C.",
title="Sabi{\'a}: Portuguese Large Language Models",
booktitle="Intelligent Systems",
year="2023",
publisher="Springer Nature Switzerland",
address="Cham",
pages="226--240",
isbn="978-3-031-45392-2"
}