Modernbert Base Nli

🚀 Model Card for ModernBERT-base

ModernBERT is a multi - task fine - tuned model on various NLI (Natural Language Inference) tasks. It offers excellent performance in reasoning tasks, long - context reasoning, sentiment analysis, and zero - shot classification.

🚀 Quick Start

Zero - shot Classification

from transformers import pipeline
classifier = pipeline("zero-shot-classification",model="tasksource/ModernBERT-base-nli")

text = "one day I will see the world"
candidate_labels = ['travel', 'cooking', 'dancing']
classifier(text, candidate_labels)

Natural Language Inference

from transformers import pipeline
pipe = pipeline("text-classification",model="tasksource/ModernBERT-base-nli")
pipe([dict(text='there is a cat',
  text_pair='there is a black cat')]) #list of (premise,hypothesis)

✨ Features

• Multi - task Fine - tuning: Trained on a wide range of NLI tasks, including MNLI, ANLI, and many others.
• Strong Reasoning Ability: Performs better than llama 3.1 8B Instruct on ANLI and FOLIO.
• Versatile Applications: Suitable for long - context reasoning, sentiment analysis, and zero - shot classification with new labels.

📦 Installation

The installation process depends on the transformers library. You can install it via pip:

pip install transformers

💻 Usage Examples

[ZS] Zero - shot Classification Pipeline

from transformers import pipeline
classifier = pipeline("zero-shot-classification",model="tasksource/ModernBERT-base-nli")

text = "one day I will see the world"
candidate_labels = ['travel', 'cooking', 'dancing']
classifier(text, candidate_labels)

NLI training data of this model includes [label - nli](https://huggingface.co/datasets/tasksource/zero - shot - label - nli), a NLI dataset specially constructed to improve this kind of zero - shot classification.

[NLI] Natural Language Inference Pipeline

from transformers import pipeline
pipe = pipeline("text-classification",model="tasksource/ModernBERT-base-nli")
pipe([dict(text='there is a cat',
  text_pair='there is a black cat')]) #list of (premise,hypothesis)

Backbone for Further Fine - tuning

This checkpoint has stronger reasoning and fine - grained abilities than the base version and can be used for further fine - tuning.

📚 Documentation

Model Details

ModernBERT is multi - task fine - tuned on tasksource NLI tasks (including MNLI, ANLI, SICK, WANLI, doc - nli, LingNLI, FOLIO, FOL - NLI, LogicNLI, Label - NLI and all datasets in the below table). It is the equivalent of an "instruct" version. The model was trained for 200k steps on an Nvidia A30 GPU.

Test Accuracy

The following table shows model test accuracy. These are the scores for the same single transformer with different classification heads on top. Further gains can be obtained by fine - tuning on a single - task, e.g., SST, but this checkpoint is great for zero - shot classification and natural language inference (contradiction/entailment/neutral classification).

📄 License

This model is licensed under the apache - 2.0 license.

📖 Citation

@inproceedings{sileo-2024-tasksource,
    title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
    author = "Sileo, Damien",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1361",
    pages = "15655--15684",
}