Modernbert Large Nli

🚀 Model Card for ModernBERT-large-nli

This model is a multi - task fine - tuned version of ModernBERT on various NLI tasks. It excels in reasoning tasks, long - context reasoning, sentiment analysis, and zero - shot classification, offering high performance and versatility.

🚀 Quick Start

This model is a multi - task fine - tuned version of ModernBERT on tasksource NLI tasks (including MNLI, ANLI, SICK, WANLI, doc - nli, LingNLI, FOLIO, FOL - NLI, LogicNLI, Label - NLI, and all datasets in the table below). It's an "instruct" equivalent. The model was trained for 200k steps on an Nvidia A30 GPU.

It performs extremely well in reasoning tasks (better than llama 3.1 8B Instruct on ANLI and FOLIO), long - context reasoning, sentiment analysis, and zero - shot classification with new labels.

The following table shows the model's test accuracy. These scores are for the same single transformer with different classification heads on top. Further improvements can be achieved by fine - tuning on a single task, e.g., SST. However, this checkpoint is excellent for zero - shot classification and natural language inference (contradiction/entailment/neutral classification).

💻 Usage Examples

Basic Usage

[ZS] Zero - shot classification pipeline

from transformers import pipeline
classifier = pipeline("zero-shot-classification",model="tasksource/ModernBERT-large-nli")

text = "one day I will see the world"
candidate_labels = ['travel', 'cooking', 'dancing']
classifier(text, candidate_labels)

The NLI training data of this model includes label - nli, a NLI dataset specially constructed to improve this kind of zero - shot classification.

[NLI] Natural language inference pipeline

from transformers import pipeline
pipe = pipeline("text-classification",model="tasksource/ModernBERT-large-nli")
pipe([dict(text='there is a cat',
  text_pair='there is a black cat')]) #list of (premise,hypothesis)

Advanced Usage

Backbone for further fine - tuning

This checkpoint has stronger reasoning and fine - grained abilities than the base version and can be used for further fine - tuning.

📄 License

This model is licensed under the apache - 2.0 license.

📚 Documentation

Citation

@inproceedings{sileo-2024-tasksource,
    title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
    author = "Sileo, Damien",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1361",
    pages = "15655--15684",
}