Dpr Question Encoder Single Lfqa Wiki

🚀 Question Encoder Model for Long - Form QA

A question encoder model based on the DPR architecture for long - form question answering, leveraging transformer pooler outputs for question representation.

🚀 Quick Start

The question encoder model is based on the DPRQuestionEncoder architecture. It uses the transformer's pooler outputs as question representations. For more details, refer to this blog post.

✨ Features

• Two - stage Training: The model vblagoje/dpr - question_encoder - single - lfqa - wiki is trained in two stages using FAIR's dpr - scale.
• Custom Training Data Creation: In each stage, custom DPR formatted training sets are created with positive, negative, and hard negative samples.
• Performance Metrics: It has specific performance metrics on the KILT benchmark for long - form question answering.

📦 Installation

No specific installation steps are provided in the original README.

💻 Usage Examples

Basic Usage

from transformers import DPRContextEncoder, DPRContextEncoderTokenizer

model = DPRQuestionEncoder.from_pretrained("vblagoje/dpr-question_encoder-single-lfqa-wiki").to(device)
tokenizer = AutoTokenizer.from_pretrained("vblagoje/dpr-question_encoder-single-lfqa-wiki")

input_ids = tokenizer("Why do airplanes leave contrails in the sky?", return_tensors="pt")["input_ids"]
embeddings = model(input_ids).pooler_output

📚 Documentation

Training

We trained vblagoje/dpr - question_encoder - single - lfqa - wiki using FAIR's dpr - scale in two stages.

• First Stage: We used a PAQ - based pretrained checkpoint and fine - tuned the retriever on the question - answer pairs from the LFQA dataset. Since dpr - scale requires a DPR - formatted training set input with positive, negative, and hard negative samples, we created a training file where an answer was the positive sample, negatives were answers unrelated to the question, and hard negative samples were chosen from answers on questions with a cosine similarity between 0.55 and 0.65.
• Second Stage: We created a new DPR training set using positives, negatives, and hard negatives from the Wikipedia/Faiss index created in the first stage instead of LFQA dataset answers. More precisely, for each dataset question, we queried the first - stage Wikipedia Faiss index and then used an SBert cross - encoder to score question/answer (passage) pairs with topk = 50. The cross - encoder selected the positive passage with the highest score, while the bottom seven answers were selected for hard - negatives. Negative samples were again chosen to be answers unrelated to a given dataset question. After creating a DPR - formatted training file with Wikipedia - sourced positive, negative, and hard negative passages, we trained DPR - based question/passage encoders using dpr - scale.

Performance

The LFQA DPR - based retriever (vblagoje/dpr - question_encoder - single - lfqa - wiki and vblagoje/dpr - ctx_encoder - single - lfqa - wiki) slightly underperforms the 'state - of - the - art' Krishna et al. "Hurdles to Progress in Long - form Question Answering" REALM - based retriever with a KILT benchmark performance of 11.2 for R - precision and 19.5 for Recall@5.

🔧 Technical Details

The model uses the DPRQuestionEncoder architecture. It takes advantage of the transformer's pooler outputs for representing questions. The training process involves complex data preparation with positive, negative, and hard negative samples, and uses FAIR's dpr - scale for training.

📄 License

This project is licensed under the MIT license.

👨‍💻 Author

• Vladimir Blagojevic: dovlex [at] gmail.com Twitter | LinkedIn