Albert Xlarge Finetuned

🚀 ALBERT-XLARGE-V2 Fine-tuned on SQuAD V2

This project presents an ALBERT-XLARGE-V2 model fine-tuned on the SQuAD V2 dataset, enabling high - performance question - answering.

🚀 Quick Start

The model is based on albert-xlarge-v2, fine - tuned on SQuAD V2 using run_squad.py.

📦 Installation

No specific installation steps are provided in the original README. However, you can refer to the official Hugging Face documentation for the basic setup of the transformers library and the necessary dependencies.

🔧 Technical Details

Training Parameters

The model was trained on 4 NVIDIA GeForce RTX 2080 Ti 11Gb. Here is the training command:

BASE_MODEL=albert-xlarge-v2
python run_squad.py \
  --version_2_with_negative \
  --model_type albert \
  --model_name_or_path $BASE_MODEL \
  --output_dir $OUTPUT_MODEL \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v2.0.json \
  --predict_file $SQUAD_DIR/dev-v2.0.json \
  --per_gpu_train_batch_size 3 \
  --per_gpu_eval_batch_size 64 \
  --learning_rate 3e-5 \
  --num_train_epochs 3.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --save_steps 2000 \
  --threads 24 \
  --warmup_steps 814 \
  --gradient_accumulation_steps 4 \
  --fp16 \
  --do_train

📚 Documentation

Evaluation

The evaluation was conducted on the dev set without sweeping for the best threshold. The following table shows the evaluation results:

💻 Usage Examples

Basic Usage

See huggingface documentation. Training on SQuAD V2 allows the model to score if a paragraph contains an answer:

start_scores, end_scores = model(input_ids) 
span_scores = start_scores.softmax(dim=1).log()[:,:,None] + end_scores.softmax(dim=1).log()[:,None,:]
ignore_score = span_scores[:,0,0] #no answer scores