Mbarthez

🚀 BARThez and mBARThez: French Sequence-to-Sequence Pretrained Models

BARThez and mBARThez are French sequence-to-sequence pretrained models based on BART, well-suited for generative tasks.

🚀 Quick Start

BARThez is a French sequence-to-sequence pretrained model based on BART. It is pretrained by learning to reconstruct a corrupted input sentence, using a 66GB corpus of French raw text for the pretraining.

Unlike existing BERT-based French language models like CamemBERT and FlauBERT, BARThez is particularly well-suited for generative tasks (such as abstractive summarization) because both its encoder and decoder are pretrained.

In addition to BARThez, which is pretrained from scratch, we continued the pretraining of a multilingual BART mBART, which improved its performance in both discriminative and generative tasks. We call the French-adapted version mBARThez.

📚 Documentation

Model Comparison

References

• Paper: https://arxiv.org/abs/2010.12321
• GitHub: https://github.com/moussaKam/BARThez

BibTeX Citation

@article{eddine2020barthez,
  title={BARThez: a Skilled Pretrained French Sequence-to-Sequence Model},
  author={Eddine, Moussa Kamal and Tixier, Antoine J-P and Vazirgiannis, Michalis},
  journal={arXiv preprint arXiv:2010.12321},
  year={2020}
}

📄 License

This project is licensed under the Apache-2.0 license.