Scibert Scivocab Uncased Squad V2

🚀 SciBERT Fine-tuned on SQuAD V2

This project presents a fine-tuned version of the allenai/scibert_scivocab_uncased model on the SQuAD V2 dataset. It aims to enhance question-answering capabilities in scientific contexts.

✨ Features

• Fine-tuned on the SQuAD V2 dataset for improved question-answering performance.
• Capable of scoring whether a paragraph contains an answer.

📦 Installation

The installation process involves fine-tuning the model using the provided script. Ensure you have the necessary dependencies and hardware.

Training Environment

Trained on 4 NVIDIA GeForce RTX 2080 Ti 11Gb.

Training Command

BASE_MODEL=allenai/scibert_scivocab_uncased
python run_squad.py \
  --version_2_with_negative \
  --model_type albert \
  --model_name_or_path $BASE_MODEL \
  --output_dir $OUTPUT_MODEL \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v2.0.json \
  --predict_file $SQUAD_DIR/dev-v2.0.json \
  --per_gpu_train_batch_size 18 \
  --per_gpu_eval_batch_size 64 \
  --learning_rate 3e-5 \
  --num_train_epochs 3.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --save_steps 2000 \
  --threads 24 \
  --warmup_steps 550 \
  --gradient_accumulation_steps 1 \
  --fp16 \
  --logging_steps 50 \
  --do_train

🔧 Technical Details

The model is fine-tuned using the run_squad.py script from the Hugging Face Transformers library. The training parameters are carefully selected to optimize the model's performance on the SQuAD V2 dataset.

📚 Documentation

Evaluation

The model is evaluated on the dev set. The following table shows the evaluation results:

Usage

For more information on using the model, refer to the huggingface documentation. Training on SQuAD V2 enables the model to score if a paragraph contains an answer.

Basic Usage

start_scores, end_scores = model(input_ids) 
span_scores = start_scores.softmax(dim=1).log()[:,:,None] + end_scores.softmax(dim=1).log()[:,None,:]
ignore_score = span_scores[:,0,0] #no answer scores