ModernBERT-large-zeroshot-v2.0 Open-Source Zero-Shot Classifier - An Efficient and Low-Memory Assistant for Multi-Text Classification

Modernbert Large Zeroshot V2.0

Developed by MoritzLaurer

A zero-shot classifier fine-tuned based on ModernBERT-large, efficient and fast with low memory usage, suitable for various text classification tasks.

Large Language Model

Transformers

Open Source License:Apache-2.0 #Zero-shot classification #Efficient inference #Multi-task adaptation

Downloads 25.66k

Release Time : 12/27/2024

Model Overview

This model is fine-tuned based on ModernBERT-large, trained on the same dataset as the zeroshot-v2.0 model in the Zeroshot classifier collection, suitable for zero-shot text classification tasks.

Model Features

Efficient and fast

The model runs extremely fast with low memory usage, offering several times speed improvement and reduced memory consumption compared to DeBERTav3, supporting larger batch processing.

Performance

Slightly underperforms DeBERTav3 on average across various test tasks but offers higher overall efficiency.

Future optimizations

Plans to leverage higher-quality synthetic data to fully utilize the 8k context window advantage and update the training composition.

Model Capabilities

Zero-shot text classification

Multi-task text classification

Efficient inference

Use Cases

Text classification

Sentiment analysis

Classify sentiment tendencies in movie reviews, product reviews, etc.

Achieved an accuracy of 0.899 on the rottentomatoes dataset.

Topic classification

Classify topics in news articles, social media content, etc.

Achieved an accuracy of 0.899 on the agnews dataset.

Hate speech detection

Detect hate speech and offensive content in text.

Achieved an accuracy of 0.814 on the hatexplain dataset.

🚀 ModernBERT-base-zeroshot-v2.0

This model is a fine - tuned version of answerdotai/ModernBERT-large on the same dataset mix as the zeroshot-v2.0 models in the Zeroshot Classifiers Collection. It offers high - speed performance and excellent memory efficiency, with potential for further improvement in future versions.

✨ Features

High - speed and Memory - efficient: The model is very fast and memory efficient. It's multiple times faster and consumes multiple times less memory than DeBERTav3. The memory efficiency enables larger batch sizes. A ~2x speed increase can be achieved by enabling bf16 (instead of fp16).
Room for Improvement: It performs slightly worse than DeBERTav3 on average on the tested tasks.
Future Updates: The author is preparing a newer version trained on better synthetic data to make full use of the 8k context window and update the training mix of the older zeroshot-v2.0 models.

📚 Documentation

Training results

Datasets	Mean	Mean w/o NLI	mnli_m	mnli_mm	fevernli	anli_r1	anli_r2	anli_r3	wanli	lingnli	wellformedquery	rottentomatoes	amazonpolarity	imdb	yelpreviews	hatexplain	massive	banking77	emotiondair	emocontext	empathetic	agnews	yahootopics	biasframes_sex	biasframes_offensive	biasframes_intent	financialphrasebank	appreviews	hateoffensive	trueteacher	spam	wikitoxic_toxicaggregated	wikitoxic_obscene	wikitoxic_identityhate	wikitoxic_threat	wikitoxic_insult	manifesto	capsotu
Accuracy	0.85	0.851	0.942	0.944	0.894	0.812	0.717	0.716	0.836	0.909	0.815	0.899	0.964	0.951	0.984	0.814	0.8	0.744	0.752	0.802	0.544	0.899	0.735	0.934	0.864	0.877	0.913	0.953	0.921	0.821	0.989	0.901	0.927	0.931	0.959	0.911	0.497	0.73
F1 macro	0.834	0.835	0.935	0.938	0.882	0.795	0.688	0.676	0.823	0.898	0.814	0.899	0.964	0.951	0.984	0.77	0.753	0.763	0.69	0.805	0.533	0.899	0.729	0.925	0.864	0.877	0.901	0.953	0.855	0.821	0.983	0.901	0.927	0.931	0.952	0.911	0.362	0.662
Inference text/sec (A100 40GB GPU, batch = 32)	1116.0	1104.0	1039.0	1241.0	1138.0	1102.0	1124.0	1133.0	1251.0	1240.0	1263.0	1231.0	1054.0	559.0	795.0	1238.0	1312.0	1285.0	1273.0	1268.0	992.0	1222.0	894.0	1176.0	1194.0	1197.0	1206.0	1166.0	1227.0	541.0	1199.0	1045.0	1054.0	1020.0	1005.0	1063.0	1214.0	1220.0

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 9e - 06
train_batch_size: 16
eval_batch_size: 32
seed: 42
gradient_accumulation_steps: 2
total_train_batch_size: 32
optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon = 1e - 08 and optimizer_args = No additional optimizer arguments
lr_scheduler_type: linear
lr_scheduler_warmup_ratio: 0.06
num_epochs: 2

Framework versions

Transformers 4.48.0.dev0
Pytorch 2.5.1+cu124
Datasets 3.2.0
Tokenizers 0.21.0

📄 License

The project is licensed under the Apache - 2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご