Modernbert QnA Base Squad

🚀 ModernBERT-QnA-base-squad

This repository hosts the fine-tuned ModernBERT model for Question Answering tasks, achieving impressive performance on the SQuAD dataset, making it an excellent choice for extractive question-answering applications.

🚀 Quick Start

The following example demonstrates how to use the fine-tuned model for question answering using the Hugging Face pipeline.

For now, you have to install a specific transformers fork, until the official PR will be merged.

> pip uninstall transformers -y
> git clone https://github.com/bakrianoo/transformers.git
> cd transformers && git checkout feat-ModernBert-QnA-Support && pip install -e .

💻 Usage Examples

Basic Usage

from transformers.models.modernbert.modular_modernbert import ModernBertForQuestionAnswering
from transformers import AutoTokenizer, pipeline

# Load the model and tokenizer
model_id = "rankyx/ModernBERT-QnA-base-squad"
model = ModernBertForQuestionAnswering.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Initialize the question-answering pipeline
question_answerer = pipeline("question-answering", model=model, tokenizer=tokenizer)

# Example input
question = "How many parameters does BLOOM contain?"
context = "BLOOM has 176 billion parameters and can generate text in 46 natural languages and 13 programming languages."

# Get the answer
result = question_answerer(question=question, context=context)
print(result)

Advanced Usage

from transformers import pipeline

model_id = "rankyx/ModernBERT-QnA-base-squad"
question_answerer = pipeline("question-answering", model=model_id)

# Input
question = "Describe the primary focus of the United Nations."
context = """
The United Nations (UN) is an international organization founded in 1945. It is currently made up of 193 Member States. The mission and work of the United Nations are guided by the purposes and principles contained in its founding Charter. The UN is best known for its peacekeeping, peacebuilding, conflict prevention, and humanitarian assistance. It also works on promoting sustainable development, protecting human rights, upholding international law, and delivering humanitarian aid. Through its various specialized agencies, funds, and programs, the UN addresses issues ranging from health to education to climate change.
"""

# Get the answer
result = question_answerer(question=question, context=context)
print(result)

✨ Features

• Model ID: rankyx/ModernBERT-QnA-base-squad
• Base Model: answerdotai/ModernBERT-base
• Dataset: SQuAD
• Evaluation Metrics:
  • F1 Score: 92.59
  • Exact Match: 86.45
• Training Framework: Hugging Face Transformers

Read more about ModernBERT's capabilities in this Hugging Face blog post.

📦 Installation

python run_qa.py \
  --model_name_or_path "answerdotai/ModernBERT-base" \
  --dataset_name squad \
  --do_train \
  --do_eval \
  --overwrite_output_dir \
  --per_device_train_batch_size 25 \
  --per_device_eval_batch_size 20 \
  --eval_strategy="steps" \
  --save_strategy="epoch" \
  --logging_steps 50 \
  --eval_steps 500 \
  --learning_rate 3e-5 \
  --warmup_ratio 0.1 \
  --weight_decay 0.01 \
  --doc_stride 128 \
  --max_seq_length 384 \
  --max_answer_length 128 \
  --num_train_epochs 2 \
  --run_name="ModernBERT-QnA-base-squad" \
  --output_dir="/path/to/output/directory"

If you have multiple GPUs and getting the error: RuntimeError: Detected that you are using FX to symbolically trace a dynamo-optimized function try this:

accelerate launch run_qa.py \
...other parameters

📚 Documentation

Model Overview

The ModernBERT-QnA-base-squad is a fine-tuned version of the ModernBERT model, specifically tailored for question-answering tasks. It has been trained on the SQuAD dataset and shows excellent performance in extractive question-answering applications.

Fine-tuning Process

The model was fine-tuned using the Hugging Face Transformers library with the official script for question answering.

🔧 Technical Details

The model uses the ModernBertForQuestionAnswering architecture from the transformers library. During fine-tuning, it was trained with specific hyperparameters such as a learning rate of 3e-5, a warmup ratio of 0.1, and a weight decay of 0.01.

📄 License

This model is licensed under the Apache 2.0 License. See LICENSE for details.

📊 Results

Evaluation Metrics

• F1 Score: 92.59
• Exact Match: 86.45
• Training Loss: 0.860

✍️ Citation

If you use this model in your research, please cite it as follows:

@misc{rankyx2024modernbertqna,
  title={ModernBERT-QnA-base-squad},
  author={Abu Bakr},
  year={2024},
  howpublished={\url{https://huggingface.co/rankyx/ModernBERT-QnA-base-squad}}
}

🙌 Acknowledgments

Special thanks to the Hugging Face team for providing the tools and resources to fine-tune and deploy this model.

🤝 Feedback and Contributions

We welcome feedback and contributions! Please feel free to open an issue or submit a pull request.

⚠️ Important Note

For now, you have to install a specific transformers fork, until the official PR will be merged.

💡 Usage Tip

If you have multiple GPUs and getting the error: RuntimeError: Detected that you are using FX to symbolically trace a dynamo-optimized function, try using accelerate launch run_qa.py instead.