Deberta V3 Large Zeroshot V1.1 All 33

🚀 Deberta-v3-large-zeroshot-v1.1-all-33

This model is designed for zero-shot classification using the Hugging Face pipeline. It can solve a universal classification task: determining whether a hypothesis is "true" or "not true" given a text (entailment vs. not_entailment). This task format is based on the Natural Language Inference (NLI) task, and any classification task can be reformulated into it.

A detailed description of how the model was trained and how to use it can be found in this paper.

✨ Features

• Designed for zero-shot classification with the Hugging Face pipeline.
• Can handle a universal classification task based on NLI.
• Trained on a mixture of 33 datasets and 387 classes.

📦 Installation

To use the model, you need to install the transformers library with sentencepiece:

#!pip install transformers[sentencepiece]

💻 Usage Examples

Basic Usage

#!pip install transformers[sentencepiece]
from transformers import pipeline
text = "Angela Merkel is a politician in Germany and leader of the CDU"
hypothesis_template = "This example is about {}"
classes_verbalized = ["politics", "economy", "entertainment", "environment"]
zeroshot_classifier = pipeline("zero-shot-classification", model="MoritzLaurer/deberta-v3-large-zeroshot-v1.1-all-33")
output = zeroshot_classifier(text, classes_verbalized, hypothesis_template=hypothesis_template, multi_label=False)
print(output)

Advanced Usage

from transformers import pipeline
text = "Angela Merkel is a politician in Germany and leader of the CDU"
hypothesis_template = "Merkel is the leader of the party: {}"
classes_verbalized = ["CDU", "SPD", "Greens"]
zeroshot_classifier = pipeline("zero-shot-classification", model="MoritzLaurer/deberta-v3-large-zeroshot-v1.1-all-33")
output = zeroshot_classifier(text, classes_verbalized, hypothesis_template=hypothesis_template, multi_label=False)
print(output)

📚 Documentation

Training Data

The model was trained on a mixture of 33 datasets and 387 classes reformatted into the universal NLI format:

1. Five NLI datasets with ~885k texts: "mnli", "anli", "fever", "wanli", "ling".
2. 28 classification tasks reformatted into the universal NLI format. ~51k cleaned texts were used to avoid overfitting:
   • 'amazonpolarity', 'imdb', 'appreviews', 'yelpreviews', 'rottentomatoes',
   • 'emotiondair', 'emocontext', 'empathetic',
   • 'financialphrasebank', 'banking77', 'massive',
   • 'wikitoxic_toxicaggregated', 'wikitoxic_obscene', 'wikitoxic_threat', 'wikitoxic_insult', 'wikitoxic_identityhate',
   • 'hateoffensive', 'hatexplain', 'biasframes_offensive', 'biasframes_sex', 'biasframes_intent',
   • 'agnews', 'yahootopics',
   • 'trueteacher', 'spam', 'wellformedquery',
   • 'manifesto', 'capsotu'.

See details on each dataset here: https://github.com/MoritzLaurer/zeroshot-classifier/blob/main/datasets_overview.csv

Note that compared to other NLI models, this model predicts two classes (entailment vs. not_entailment) instead of three (entailment/neutral/contradiction). The model was only trained on English data. For multilingual use-cases, it is recommended to machine translate texts to English with libraries like EasyNMT.

Details on Data and Training

The code for preparing the data and training & evaluating the model is fully open-source here: https://github.com/MoritzLaurer/zeroshot-classifier/tree/main

Hyperparameters and other details are available in this Weights & Biases repo: https://wandb.ai/moritzlaurer/deberta-v3-large-zeroshot-v1-1-all-33/table?workspace=user-

Metrics

Balanced accuracy is reported for all datasets. deberta-v3-large-zeroshot-v1.1-all-33 was trained on all datasets, with only a maximum of 500 texts per class to avoid overfitting. The metrics on these datasets are not strictly zeroshot, as the model has seen some data for each task during training. deberta-v3-large-zeroshot-v1.1-heldout indicates zeroshot performance on the respective dataset.

To calculate these zeroshot metrics, the pipeline was run 28 times, each time with one dataset held out from training to simulate a zeroshot setup.

🔧 Technical Details

The model is based on the DeBERTa-v3-large architecture and is trained on a mixture of NLI and classification datasets reformatted into the universal NLI format. It predicts two classes (entailment vs. not_entailment) instead of three, which is different from other NLI models.

📄 License

The base model (DeBERTa-v3) is published under the MIT license. The datasets the model was fine-tuned on are published under a diverse set of licenses. See details here: https://github.com/MoritzLaurer/zeroshot-classifier/blob/main/datasets_overview.csv

Citation

If you use this model academically, please cite:

@misc{laurer_building_2023,
	title = {Building {Efficient} {Universal} {Classifiers} with {Natural} {Language} {Inference}},
	url = {http://arxiv.org/abs/2312.17543},
	doi = {10.48550/arXiv.2312.17543},
	abstract = {Generative Large Language Models (LLMs) have become the mainstream choice for fewshot and zeroshot learning thanks to the universality of text generation. Many users, however, do not need the broad capabilities of generative LLMs when they only want to automate a classification task. Smaller BERT-like models can also learn universal tasks, which allow them to do any text classification task without requiring fine-tuning (zeroshot classification) or to learn new tasks with only a few examples (fewshot), while being significantly more efficient than generative LLMs. This paper (1) explains how Natural Language Inference (NLI) can be used as a universal classification task that follows similar principles as instruction fine-tuning of generative LLMs, (2) provides a step-by-step guide with reusable Jupyter notebooks for building a universal classifier, and (3) shares the resulting universal classifier that is trained on 33 datasets with 389 diverse classes. Parts of the code we share has been used to train our older zeroshot classifiers that have been downloaded more than 55 million times via the Hugging Face Hub as of December 2023. Our new classifier improves zeroshot performance by 9.4\%.},
	urldate = {2024-01-05},
	publisher = {arXiv},
	author = {Laurer, Moritz and van Atteveldt, Wouter and Casas, Andreu and Welbers, Kasper},
	month = dec,
	year = {2023},
	note = {arXiv:2312.17543 [cs]},
	keywords = {Computer Science - Artificial Intelligence, Computer Science - Computation and Language},
}

⚠️ Important Note

DeBERTa-v3 was released on 06.12.21 and older versions of HF Transformers can have issues running the model (e.g., resulting in an issue with the tokenizer). Using Transformers>=4.13 might solve some issues.

💡 Usage Tip

For multilingual use-cases, you can machine translate texts to English with libraries like EasyNMT.

Ideas for cooperation or questions?

If you have questions or ideas for cooperation, contact me at m{dot}laurer{at}vu{dot}nl or LinkedIn