Camembert Bio Base

🚀 CamemBERT-bio : a Tasty French Language Model Better for your Health

CamemBERT-bio is a state-of-the-art French biomedical language model. It's built via continual-pretraining from camembert-base. Trained on a 413M-word French public biomedical corpus with scientific documents, drug leaflets, and clinical cases, it outperforms camembert-base by an average of 2.54 points in F1 score on 5 different biomedical named entity recognition tasks.

✨ Features

• Developed by Rian Touchent and Eric Villemonte de La Clergerie.
• Logo designed by Alix Chagué.
• Licensed under the MIT license.

📚 Documentation

Abstract

Clinical data in hospitals are increasingly accessible for research through clinical data warehouses. However, these documents are unstructured, so it's necessary to extract information from medical reports for clinical studies. Transfer learning with BERT-like models like CamemBERT has made significant progress, especially in named entity recognition. But these models are trained for plain language and are less effective on biomedical data. That's why we propose a new French public biomedical dataset and continue the pre-training of CamemBERT. Thus, we introduce the first version of CamemBERT-bio, a specialized public model for the French biomedical domain, which shows an average improvement of 2.54 points in F1 score on different biomedical named entity recognition tasks.

🔧 Technical Details

Training Details

Training Data

Training Procedure

We used continual-pretraining from camembert-base. The model was trained using the Masked Language Modeling (MLM) objective with Whole Word Masking for 50k steps over 39 hours on 2 Tesla V100s.

Evaluation

Fine-tuning

For fine-tuning, we used Optuna to select hyperparameters. The learning rate was set to 5e-5, with a warmup ratio of 0.224 and a batch size of 16. The fine-tuning process lasted for 2000 steps. A simple linear layer was added on top of the model for prediction, and none of the CamemBERT layers were frozen during fine-tuning.

Scoring

We used the seqeval tool in strict mode with the IOB2 scheme to evaluate the model's performance. For each evaluation, the best fine-tuned model on the validation set was selected to calculate the final score on the test set. To ensure reliability, we averaged the results over 10 evaluations with different seeds.

Results

Environmental Impact estimation

• Hardware Type: 2 x Tesla V100
• Hours used: 39 hours
• Provider: INRIA clusters
• Compute Region: Paris, France
• Carbon Emitted: 0.84 kg CO2 eq.

📄 License

This project is licensed under the MIT license.

📖 Citation information

@inproceedings{touchent-de-la-clergerie-2024-camembert-bio,
    title = "{C}amem{BERT}-bio: Leveraging Continual Pre-training for Cost-Effective Models on {F}rench Biomedical Data",
    author = "Touchent, Rian  and
      de la Clergerie, {\'E}ric",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.241",
    pages = "2692--2701",
    abstract = "Clinical data in hospitals are increasingly accessible for research through clinical data warehouses. However these documents are unstructured and it is therefore necessary to extract information from medical reports to conduct clinical studies. Transfer learning with BERT-like models such as CamemBERT has allowed major advances for French, especially for named entity recognition. However, these models are trained for plain language and are less efficient on biomedical data. Addressing this gap, we introduce CamemBERT-bio, a dedicated French biomedical model derived from a new public French biomedical dataset. Through continual pre-training of the original CamemBERT, CamemBERT-bio achieves an improvement of 2.54 points of F1-score on average across various biomedical named entity recognition tasks, reinforcing the potential of continual pre-training as an equally proficient yet less computationally intensive alternative to training from scratch. Additionally, we highlight the importance of using a standard evaluation protocol that provides a clear view of the current state-of-the-art for French biomedical models.",
}

@inproceedings{touchent:hal-04130187,
  TITLE = {{CamemBERT-bio : Un mod{\`e}le de langue fran{\c c}ais savoureux et meilleur pour la sant{\'e}}},
  AUTHOR = {Touchent, Rian and Romary, Laurent and De La Clergerie, Eric},
  URL = {https://hal.science/hal-04130187},
  BOOKTITLE = {{18e  Conf{\'e}rence en Recherche d'Information et Applications \\ 16e Rencontres Jeunes Chercheurs en RI \\ 30e Conf{\'e}rence sur le Traitement Automatique des Langues Naturelles \\ 25e Rencontre des {\'E}tudiants Chercheurs en Informatique pour le Traitement Automatique des Langues}},
  ADDRESS = {Paris, France},
  EDITOR = {Servan, Christophe and Vilnat, Anne},
  PUBLISHER = {{ATALA}},
  PAGES = {323-334},
  YEAR = {2023},
  KEYWORDS = {comptes rendus m{\'e}dicaux ; TAL clinique ; CamemBERT ; extraction d'information ; biom{\'e}dical ; reconnaissance d'entit{\'e}s nomm{\'e}es},
  HAL_ID = {hal-04130187},
  HAL_VERSION = {v1},
}