Camembertav2 Base

🚀 CamemBERT(a)-v2: A Smarter French Language Model Aged to Perfection

CamemBERTav2 is a French language model pretrained on a large corpus of 275B tokens of French text. It's based on the DebertaV2 architecture and is the second version of the CamemBERTa model. Trained using the Replaced Token Detection (RTD) objective with a 20% mask rate on 275B tokens across 32 H100 GPUs, the training dataset combines French OSCAR dumps from the CulturaX Project, French scientific documents from HALvest, and the French Wikipedia.

This model can directly replace the original CamemBERTa model. Note that the new tokenizer differs from the original CamemBERTa tokenizer, so you'll need to use Fast Tokenizers to utilize the model. It's compatible with DebertaV2TokenizerFast from the transformers library, even though the original DebertaV2TokenizerFast was sentencepiece - based.

✨ Features

• Larger Pretraining Dataset: The new version uses a much larger pretraining dataset with 275B unique tokens, compared to the previous ~32B.
• New Tokenizer: A newly built tokenizer based on WordPiece with 32,768 tokens. It adds newline and tab characters, supports emojis, and better handles numbers by splitting them into two - digit tokens.
• Extended Context Window: It has an extended context window of 1024 tokens.

For more details, refer to the CamemBERTv2 paper.

📦 Installation

This section doesn't explicitly mention installation steps, so it's skipped.

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModel, AutoModelForMaskedLM

camembertav2 = AutoModel.from_pretrained("almanach/camembertav2-base")
tokenizer = AutoTokenizer.from_pretrained("almanach/camembertav2-base")

📚 Documentation

Fine - tuning Results

Datasets used for fine - tuning include POS tagging and Dependency Parsing (GSD, Rhapsodie, Sequoia, FSMB), NER (FTB), the FLUE benchmark (XNLI, CLS, PAWS - X), the French Question Answering Dataset (FQuAD), Social Media NER (Counter - NER), and Medical NER (CAS1, CAS2, E3C, EMEA, MEDLINE).

Finetuned models are available in the following collection: CamemBERTav2 Finetuned Models

Pretraining Codebase

We use the pretraining codebase from the CamemBERTa repository for all v2 models.

📄 License

The model is released under the MIT license.

📖 Citation

@misc{antoun2024camembert20smarterfrench,
      title={CamemBERT 2.0: A Smarter French Language Model Aged to Perfection},
      author={Wissam Antoun and Francis Kulumba and Rian Touchent and Éric de la Clergerie and Benoît Sagot and Djamé Seddah},
      year={2024},
      eprint={2411.08868},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2411.08868},
}