Legal T5 Small Multitask Sv It

🚀 legal_t5_small_multitask_sv_it model

A model for translating legal text from Swedish to Italian, offering efficient and accurate legal language translation services.

🚀 Quick Start

The legal_t5_small_multitask_sv_it model is designed for translating legal text from Swedish to Italian. It was first released in this repository. The model is trained on three parallel corpora with 42 language pairs from JRC-ACQUIS, Europarl, and DCEP, along with an unsupervised task similar to a masked language model prediction task.

💻 Usage Examples

Basic Usage

Here is how to use this model to translate legal text from Swedish to Italian in PyTorch:

from transformers import AutoTokenizer, AutoModelWithLMHead, TranslationPipeline

pipeline = TranslationPipeline(
model=AutoModelWithLMHead.from_pretrained("SEBIS/legal_t5_small_multitask_sv_it"),
tokenizer=AutoTokenizer.from_pretrained(pretrained_model_name_or_path = "SEBIS/legal_t5_small_multitask_sv_it", do_lower_case=False, 
                                            skip_special_tokens=True),
    device=0
)

sv_text = "De nationella tillsynsmyndigheterna får använda"

pipeline([sv_text], max_length=512)

✨ Features

• Multitask Learning: No pretraining is involved. Instead, an unsupervised task is added to all translation tasks to achieve a multitask learning scenario.
• Parallel Training: Trained on three parallel corpora with 42 language pairs from JRC-ACQUIS, Europarl, and DCEP.

📦 Installation

No specific installation steps are provided in the original document.

📚 Documentation

Model description

No pretraining is involved in case of legal_t5_small_multitask_sv_it model, rather the unsupervised task is added with all the translation task to realize the multitask learning scenario.

Intended uses & limitations

The model could be used for translation of legal texts from Swedish to Italian.

Training data

The legal_t5_small_multitask_sv_it model (the supervised task which involved only the corresponding language pair and as well as unsupervised task where all of the data of all language pairs were available) model was trained on JRC-ACQUIS, EUROPARL, and DCEP dataset consisting of 8 Million parallel texts.

Training procedure

The model was trained on a single TPU Pod V3-8 for 250K steps in total, using sequence length 512 (batch size 4096). It has a total of approximately 220M parameters and was trained using the encoder-decoder architecture. The optimizer used is AdaFactor with inverse square root learning rate schedule.

Preprocessing

An unigram model trained with 88M lines of text from the parallel corpus (of all possible language pairs) to get the vocabulary (with byte pair encoding), which is used with this model.

Evaluation results

When the model is used for translation test dataset, achieves the following results:

BibTeX entry and citation info

Created by Ahmed Elnaggar/@Elnaggar_AI | LinkedIn

🔧 Technical Details

The model is trained on a single TPU Pod V3-8 for 250K steps in total, using sequence length 512 (batch size 4096). It has a total of approximately 220M parameters and is trained using the encoder-decoder architecture. The optimizer used is AdaFactor with inverse square root learning rate schedule. An unigram model is trained with 88M lines of text from the parallel corpus to get the vocabulary (with byte pair encoding).

📄 License

No license information is provided in the original document.