Bioformer 8L Mnli

🚀 [bioformer-cased-v1.0]

The bioformer-cased-v1.0 model is fine-tuned on the MNLI dataset, offering enhanced performance for specific natural language processing tasks.

🚀 Quick Start

The bioformer-cased-v1.0 model has been fine-tuned on the MNLI dataset for 2 epochs.

🔧 Technical Details

The fine-tuning process was carried out on two NVIDIA GeForce GTX 1080 Ti GPUs (11GB). The specific parameters used are as follows:

max_seq_length=512
per_device_train_batch_size=16
total train batch size (w. parallel, distributed & accumulation) = 32
learning_rate=3e-5

📚 Documentation

Evaluation Results

The evaluation accuracy of the fine-tuned model is eval_accuracy = 0.803973.

Speed

In our experiments, the inference speed of Bioformer is 3 times faster than BERT-base/BioBERT/PubMedBERT, and 40% faster than DistilBERT.

More Information

The Multi-Genre Natural Language Inference Corpus is a crowdsourced collection of sentence pairs with textual entailment annotations. Given a premise sentence and a hypothesis sentence, the task is to predict whether the premise entails the hypothesis (entailment), contradicts the hypothesis (contradiction), or neither (neutral). The premise sentences are gathered from ten different sources, including transcribed speech, fiction, and government reports. The authors of the benchmark use the standard test set, for which they obtained private labels from the RTE authors, and evaluate on both the matched (in-domain) and mismatched (cross-domain) section. They also use and recommend the SNLI corpus as 550k examples of auxiliary training data. (source: https://huggingface.co/datasets/glue)