Model Overview
Model Features
Model Capabilities
Use Cases
đ ModernBERT-QnA-base-squad
This repository hosts the fine-tuned ModernBERT model for Question Answering tasks, achieving impressive performance on the SQuAD dataset, making it an excellent choice for extractive question-answering applications.
đ Quick Start
The following example demonstrates how to use the fine-tuned model for question answering using the Hugging Face pipeline
.
For now, you have to install a specific transformers
fork, until the official PR will be merged.
> pip uninstall transformers -y
> git clone https://github.com/bakrianoo/transformers.git
> cd transformers && git checkout feat-ModernBert-QnA-Support && pip install -e .
đģ Usage Examples
Basic Usage
from transformers.models.modernbert.modular_modernbert import ModernBertForQuestionAnswering
from transformers import AutoTokenizer, pipeline
# Load the model and tokenizer
model_id = "rankyx/ModernBERT-QnA-base-squad"
model = ModernBertForQuestionAnswering.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
# Initialize the question-answering pipeline
question_answerer = pipeline("question-answering", model=model, tokenizer=tokenizer)
# Example input
question = "How many parameters does BLOOM contain?"
context = "BLOOM has 176 billion parameters and can generate text in 46 natural languages and 13 programming languages."
# Get the answer
result = question_answerer(question=question, context=context)
print(result)
Advanced Usage
from transformers import pipeline
model_id = "rankyx/ModernBERT-QnA-base-squad"
question_answerer = pipeline("question-answering", model=model_id)
# Input
question = "Describe the primary focus of the United Nations."
context = """
The United Nations (UN) is an international organization founded in 1945. It is currently made up of 193 Member States. The mission and work of the United Nations are guided by the purposes and principles contained in its founding Charter. The UN is best known for its peacekeeping, peacebuilding, conflict prevention, and humanitarian assistance. It also works on promoting sustainable development, protecting human rights, upholding international law, and delivering humanitarian aid. Through its various specialized agencies, funds, and programs, the UN addresses issues ranging from health to education to climate change.
"""
# Get the answer
result = question_answerer(question=question, context=context)
print(result)
⨠Features
- Model ID:
rankyx/ModernBERT-QnA-base-squad
- Base Model: answerdotai/ModernBERT-base
- Dataset: SQuAD
- Evaluation Metrics:
- F1 Score: 92.59
- Exact Match: 86.45
- Training Framework: Hugging Face Transformers
Read more about ModernBERT's capabilities in this Hugging Face blog post.
đĻ Installation
python run_qa.py \
--model_name_or_path "answerdotai/ModernBERT-base" \
--dataset_name squad \
--do_train \
--do_eval \
--overwrite_output_dir \
--per_device_train_batch_size 25 \
--per_device_eval_batch_size 20 \
--eval_strategy="steps" \
--save_strategy="epoch" \
--logging_steps 50 \
--eval_steps 500 \
--learning_rate 3e-5 \
--warmup_ratio 0.1 \
--weight_decay 0.01 \
--doc_stride 128 \
--max_seq_length 384 \
--max_answer_length 128 \
--num_train_epochs 2 \
--run_name="ModernBERT-QnA-base-squad" \
--output_dir="/path/to/output/directory"
If you have multiple GPUs and getting the error: RuntimeError: Detected that you are using FX to symbolically trace a dynamo-optimized function
try this:
accelerate launch run_qa.py \
...other parameters
đ Documentation
Model Overview
The ModernBERT-QnA-base-squad is a fine-tuned version of the ModernBERT model, specifically tailored for question-answering tasks. It has been trained on the SQuAD dataset and shows excellent performance in extractive question-answering applications.
Fine-tuning Process
The model was fine-tuned using the Hugging Face Transformers library with the official script for question answering.
đ§ Technical Details
The model uses the ModernBertForQuestionAnswering
architecture from the transformers
library. During fine-tuning, it was trained with specific hyperparameters such as a learning rate of 3e-5
, a warmup ratio of 0.1
, and a weight decay of 0.01
.
đ License
This model is licensed under the Apache 2.0 License. See LICENSE for details.
đ Results
Evaluation Metrics
- F1 Score: 92.59
- Exact Match: 86.45
- Training Loss: 0.860
âī¸ Citation
If you use this model in your research, please cite it as follows:
@misc{rankyx2024modernbertqna,
title={ModernBERT-QnA-base-squad},
author={Abu Bakr},
year={2024},
howpublished={\url{https://huggingface.co/rankyx/ModernBERT-QnA-base-squad}}
}
đ Acknowledgments
Special thanks to the Hugging Face team for providing the tools and resources to fine-tune and deploy this model.
đ¤ Feedback and Contributions
We welcome feedback and contributions! Please feel free to open an issue or submit a pull request.
â ī¸ Important Note
For now, you have to install a specific
transformers
fork, until the official PR will be merged.
đĄ Usage Tip
If you have multiple GPUs and getting the error:
RuntimeError: Detected that you are using FX to symbolically trace a dynamo-optimized function
, try usingaccelerate launch run_qa.py
instead.
Property | Details |
---|---|
Model Type | modernbert |
Training Data | SQuAD |







