dpr-question_encoder_bert_uncased_L-2_H-128_A-2 Open Source Model

Home

Dpr Question Encoder Bert Uncased L 2 H 128 A 2

Developed by nlpconnect

DPR question encoder model based on BERT architecture for dense passage retrieval tasks

Text Embedding

Transformers

Open Source License:Apache-2.0 #Dense Passage Retrieval #Question Answering System #BERT Lightweight Version

Downloads 21

Release Time : 3/2/2022

Model Overview

This model is the question encoder component of the Dense Passage Retrieval (DPR) system based on BERT architecture, used to encode natural language questions into vector representations for similarity matching with vectors generated by the passage encoder.

Model Features

Lightweight BERT Architecture

Uses a small BERT structure with 2 layers and 128-dimensional hidden layers for high computational efficiency

Dense Retrieval Capability

Optimized for dense passage retrieval tasks, capable of effectively encoding question semantics

High Performance

Achieves 60.53% R@10 recall rate on the NQ development dataset

Model Capabilities

Natural language question encoding

Semantic vector generation

Dense retrieval support

Use Cases

Question Answering Systems

Open-Domain Question Answering

Used to build retrieval components in open-domain question answering systems

Outperforms baseline models on the NQ dataset

Information Retrieval

Document Retrieval

Used to retrieve document passages most relevant to user queries

Achieves 49.68% R@10 recall rate on BEIR test data

🚀 dpr-question_encoder_bert_uncased_L-2_H-128_A-2

This model is a question encoder based on BERT, trained from scratch on specific datasets, aiming to provide effective encoding for questions in the DPR framework.

🚀 Quick Start

This model (google/bert_uncased_L-2_H-128_A-2) was trained from scratch on the training data: data.retriever.nq-adv-hn-train(facebookresearch/DPR). It achieves the following results on the evaluation set:

✨ Features

Evaluation data

facebook-dpr-dev-dataset from official DPR github | Model Name | Data Name | Number of Queries | Number of Passages | R@10 | R@20 | R@50 | R@100 | R@100 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | nlpconnect/dpr-ctx_encoder_bert_uncased_L-2_H-128_A-2(our) | nq-dev dataset | 6445 | 199795 | 60.53% | 68.28% | 76.07% | 80.98% | 91.45% | | nlpconnect/dpr-ctx_encoder_bert_uncased_L-12_H-128_A-2(our) | nq-dev dataset | 6445 | 199795 | 65.43% | 71.99% | 79.03% | 83.24% | 92.11% | | *facebook/dpr-ctx_encoder-single-nq-base(hf/fb) | nq-dev dataset | 6445 | 199795 | 40.94% | 49.27% | 59.05% | 66.00% | 82.00% |
UKPLab/beir test data (using only the first 2lac passages) | Model Name | Data Name | Number of Queries | Number of Passages | R@10 | R@20 | R@50 | R@100 | R@100 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | nlpconnect/dpr-ctx_encoder_bert_uncased_L-2_H-128_A-2(our) | nq-test dataset | 3452 | 200001 | 49.68% | 59.06% | 69.40% | 75.75% | 89.28% | | nlpconnect/dpr-ctx_encoder_bert_uncased_L-12_H-128_A-2(our) | nq-test dataset | 3452 | 200001 | 51.62% | 61.09% | 70.10% | 76.07% | 88.70% | | *facebook/dpr-ctx_encoder-single-nq-base(hf/fb) | nq-test dataset | 3452 | 200001 | 32.93% | 43.74% | 56.95% | 66.30% | 83.92% |

Note: * means we have evaluated on the same eval dataset.

💻 Usage Examples

Basic Usage

passage_encoder = TFAutoModel.from_pretrained("nlpconnect/dpr-ctx_encoder_bert_uncased_L-12_H-128_A-2")
query_encoder = TFAutoModel.from_pretrained("nlpconnect/dpr-question_encoder_bert_uncased_L-12_H-128_A-2")

p_tokenizer = AutoTokenizer.from_pretrained("nlpconnect/dpr-ctx_encoder_bert_uncased_L-12_H-128_A-2")
q_tokenizer = AutoTokenizer.from_pretrained("nlpconnect/dpr-question_encoder_bert_uncased_L-12_H-128_A-2")

def get_title_text_combined(passage_dicts):
    res = []
    for p in passage_dicts:
        res.append(tuple((p['title'], p['text'])))
    return res
    
processed_passages = get_title_text_combined(passage_dicts)

def extracted_passage_embeddings(processed_passages, model_config):
    passage_inputs = tokenizer.batch_encode_plus(
                    processed_passages,
                    add_special_tokens=True,
                    truncation=True,
                    padding="max_length",
                    max_length=model_config.passage_max_seq_len,
                    return_token_type_ids=True
                )
    passage_embeddings = passage_encoder.predict([np.array(passage_inputs['input_ids']), 
                                                np.array(passage_inputs['attention_mask']), 
                                                np.array(passage_inputs['token_type_ids'])], 
                                                batch_size=512, 
                                                verbose=1)
    return passage_embeddings
    
passage_embeddings = extracted_passage_embeddings(processed_passages, model_config)


def extracted_query_embeddings(queries, model_config):
    query_inputs = tokenizer.batch_encode_plus(
                    queries,
                    add_special_tokens=True,
                    truncation=True,
                    padding="max_length",
                    max_length=model_config.query_max_seq_len,
                    return_token_type_ids=True
                )
    query_embeddings = query_encoder.predict([np.array(query_inputs['input_ids']), 
                                                np.array(query_inputs['attention_mask']), 
                                                np.array(query_inputs['token_type_ids'])], 
                                                batch_size=512, 
                                                verbose=1)
    return query_embeddings
    

query_embeddings = extracted_query_embeddings(queries, model_config)

🔧 Technical Details

Training hyperparameters

The following hyperparameters were used during training:

optimizer: None
training_precision: float32

Framework versions

Transformers 4.15.0
TensorFlow 2.7.0
Tokenizers 0.10.3

📄 License

This model is licensed under the Apache 2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご