🚀 (BERT base) Language modeling in the legal domain in Portuguese
legal-bert-base-cased-ptbr is a Language Model in the legal domain in Portuguese. It is built upon the BERTimbau base model using a MASK objective. This model aims to support NLP research in the legal field, as well as computer law and legal technology applications. A variety of Portuguese legal texts were utilized for training (more details below).
The large version of the model will be available soon.
✨ Features
- Domain - Specific: Tailored for the legal domain in Portuguese.
- Research - Oriented: Supports NLP research in law - related fields.
- Versatile Application: Applicable in computer law and legal technology.
📦 Installation
The installation involves loading the pre - trained model. You can use the following Python code:
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("dominguesm/legal-bert-base-cased-ptbr")
model = AutoModel.from_pretrained("dominguesm/legal-bert-base-cased-ptbr")
from transformers import pipeline
pipe = pipeline('fill-mask', "dominguesm/legal-bert-base-cased-ptbr")
📚 Documentation
Pre - training corpora
The pre - training corpora of legal-bert-base-cased-ptbr consists of:
- 61309 - Miscellaneous legal documents
- 751 - Petitions
- 682 - Sentences
- 498 - 2nd Instance Accords
- 469 - RE grievances
- 411 - Admissibility Order
The data was provided by the BRAZILIAN SUPREME FEDERAL TRIBUNAL under the terms of use: LREC 2020. The results of this project do not represent the position of the BRAZILIAN SUPREME FEDERAL TRIBUNAL, and all responsibilities lie solely with the model's author.
Use legal-bert-base-cased-ptbr variants as Language Models
| Property |
Details |
| Model Type |
Fill - Mask Language Model |
| Training Data |
Miscellaneous legal documents, petitions, sentences, 2nd Instance Accords, RE grievances, Admissibility Order from the BRAZILIAN SUPREME FEDERAL TRIBUNAL |
| Text |
Masked token |
Predictions |
| De ordem, a Secretaria Judiciária do Supremo Tribunal Federal INTIMA a parte abaixo identificada, ou quem as suas vezes fizer, do inteiro teor do(a) despacho/decisão presente nos autos (art. 270 do Código de Processo [MASK] e art 5º da Lei 11.419/2006). |
Civil |
('Civil', 0.9999), ('civil', 0.0001), ('Penal', 0.0000), ('eletrônico', 0.0000), ('2015', 0.0000) |
| 2. INTIMAÇÃO da Autarquia: 2.2 Para que apresente em Juízo, com a contestação, cópia do processo administrativo referente ao benefício [MASK] em discussão na lide |
previdenciário |
('ora', 0.9424), ('administrativo', 0.0202), ('doença', 0.0117), ('acidente', 0.0037), ('posto', 0.0036) |
| Certifico que, nesta data, os presentes autos foram remetidos ao [MASK] para processar e julgar recurso (Agravo de Instrumento). |
STF |
('Tribunal', 0.4278), ('Supremo', 0.1657), ('origem', 0.1538), ('arquivo', 0.1415), ('sistema', 0.0216) |
| TEMA: 810. Validade da correção monetária e dos juros moratórios [MASK] sobre as condenações impostas à Fazenda Pública, conforme previstos no art. 1º - F da Lei 9.494/1997, com a redação dada pela Lei 11.960/2009. |
incidentes |
('incidentes', 0.9979), ('incidente', 0.0021), ('aplicados', 0.0000), (',', 0.0000), ('aplicada', 0.0000) |
Training results
Num examples = 353435
Num Epochs = 3
Instantaneous batch size per device = 4
Total train batch size (w. parallel, distributed & accumulation) = 32
Gradient Accumulation steps = 1
Total optimization steps = 33135
TRAIN RESULTS
"epoch": 3.0
"train_loss": 0.6107781137512769
"train_runtime": 10192.1545
"train_samples": 353435
"train_samples_per_second": 104.031
"train_steps_per_second": 3.251
EVAL RESULTS
"epoch": 3.0
"eval_loss": 0.47251805663108826
"eval_runtime": 126.3026
"eval_samples": 17878
"eval_samples_per_second": 141.549
"eval_steps_per_second": 4.426
"perplexity": 1.604028145934512
Citation
@misc{domingues2022legal-bert-base-cased-ptbr,
author = {Domingues, Maicon},
title = {Language Model in the legal domain in Portuguese},
year={2022},
howpublished= {\url{https://huggingface.co/dominguesm/legal-bert-base-cased-ptbr/}}
}
📄 License
This model is licensed under the cc - by - 4.0 license.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
