Modernbert Base Zeroshot V2.0

🚀 ModernBERT-base-zeroshot-v2.0

This model is fine - tuned from answerdotai/ModernBERT-base, aiming to provide efficient and effective zero - shot classification capabilities.

🚀 Quick Start

This README provides detailed information about the ModernBERT-base-zeroshot-v2.0 model, including its description, training results, hyperparameters, and framework versions.

✨ Features

• High - speed and Memory - efficient: The model is multiple times faster and consumes multiple times less memory than DeBERTav3, enabling larger batch sizes. Enabling bf16 can provide a ~2x speed increase compared to fp16.
• Ongoing Improvement: The developer is preparing a newer version trained on better synthetic data to fully utilize the 8k context window and update the training mix.

📚 Documentation

Model description

This model is answerdotai/ModernBERT-base fine - tuned on the same dataset mix as the zeroshot-v2.0 models in the Zeroshot Classifiers Collection.

General takeaways

• The model is very fast and memory efficient. It's multiple times faster and consumes multiple times less memory than DeBERTav3. The memory efficiency enables larger batch sizes. I got a ~2x speed increase by enabling bf16 (instead of fp16).
• It performs slightly worse then DeBERTav3 on average on the tasks tested below.
• I'm in the process of preparing a newer version trained on better synthetic data to make full use of the 8k context window and to update the training mix of the older zeroshot-v2.0 models.

Training results

Per - dataset breakdown:

| Datasets | Mean | Mean w/o NLI | mnli_m | mnli_mm | fevernli | anli_r1 | anli_r2 | anli_r3 | wanli | lingnli | wellformedquery | rottentomatoes | amazonpolarity | imdb | yelpreviews | hatexplain | massive | banking77 | emotiondair | emocontext | empathetic | agnews | yahootopics | biasframes_sex | biasframes_offensive | biasframes_intent | financialphrasebank | appreviews | hateoffensive | trueteacher | spam | wikitoxic_toxicaggregated | wikitoxic_obscene | wikitoxic_identityhate | wikitoxic_threat | wikitoxic_insult | manifesto | capsotu | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | Accuracy | 0.831 | 0.835 | 0.932 | 0.936 | 0.884 | 0.763 | 0.647 | 0.657 | 0.823 | 0.889 | 0.753 | 0.864 | 0.949 | 0.935 | 0.974 | 0.798 | 0.788 | 0.727 | 0.789 | 0.793 | 0.489 | 0.893 | 0.717 | 0.927 | 0.851 | 0.859 | 0.907 | 0.952 | 0.926 | 0.726 | 0.978 | 0.912 | 0.914 | 0.93 | 0.951 | 0.906 | 0.476 | 0.708 | | F1 macro | 0.813 | 0.818 | 0.925 | 0.93 | 0.872 | 0.74 | 0.61 | 0.611 | 0.81 | 0.874 | 0.751 | 0.864 | 0.949 | 0.935 | 0.974 | 0.751 | 0.738 | 0.746 | 0.733 | 0.798 | 0.475 | 0.893 | 0.712 | 0.919 | 0.851 | 0.859 | 0.892 | 0.952 | 0.847 | 0.721 | 0.966 | 0.912 | 0.914 | 0.93 | 0.942 | 0.906 | 0.329 | 0.637 | | Inference text/sec (A100 40GB GPU, batch = 128) | 3472.0 | 3474.0 | 2338.0 | 4416.0 | 2993.0 | 2959.0 | 2904.0 | 3003.0 | 4647.0 | 4486.0 | 5032.0 | 4354.0 | 2466.0 | 1140.0 | 1582.0 | 4392.0 | 5446.0 | 5296.0 | 4904.0 | 4787.0 | 2251.0 | 4042.0 | 1884.0 | 4048.0 | 4032.0 | 4121.0 | 4275.0 | 3746.0 | 4485.0 | 1114.0 | 4322.0 | 2260.0 | 2274.0 | 2189.0 | 2085.0 | 2410.0 | 3933.0 | 4388.0 |

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e - 05
• train_batch_size: 32
• eval_batch_size: 128
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9, 0.999) and epsilon = 1e - 08 and optimizer_args = No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.06
• num_epochs: 2

Framework versions

• Transformers 4.48.0.dev0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0

📄 License

This model is released under the apache - 2.0 license.