Spar Wiki Bm25 Lexmodel Context Encoder

🚀 稀疏短語感知密集檢索模型（SPAR）上下文編碼器

本模型是SPAR論文中Wiki BM25詞彙模型（Λ）的上下文編碼器，旨在讓密集檢索器模仿稀疏檢索器的行為，提升檢索性能。

Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?
Xilun Chen, Kushal Lakhotia, Barlas Oğuz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta和Wen-tau Yih
Meta AI

關聯的GitHub倉庫地址：https://github.com/facebookresearch/dpr-scale/tree/main/spar

本模型是一個BERT-base大小的密集檢索器，在維基百科文章上進行訓練，以模仿BM25的行為。以下是可用的預訓練模型：

💻 使用示例

基礎用法

單獨使用詞彙模型（Λ）時，此模型應與關聯的查詢編碼器一起使用，類似於 DPR 模型。

import torch
from transformers import AutoTokenizer, AutoModel

# 分詞器對查詢和上下文編碼器是相同的
tokenizer = AutoTokenizer.from_pretrained('facebook/spar-wiki-bm25-lexmodel-query-encoder')
query_encoder = AutoModel.from_pretrained('facebook/spar-wiki-bm25-lexmodel-query-encoder')
context_encoder = AutoModel.from_pretrained('facebook/spar-wiki-bm25-lexmodel-context-encoder')

query = "Where was Marie Curie born?"
contexts = [
    "Maria Sklodowska, later known as Marie Curie, was born on November 7, 1867.",
    "Born in Paris on 15 May 1859, Pierre Curie was the son of Eugène Curie, a doctor of French Catholic origin from Alsace."
]

# 應用分詞器
query_input = tokenizer(query, return_tensors='pt')
ctx_input = tokenizer(contexts, padding=True, truncation=True, return_tensors='pt')

# 計算嵌入：取 [CLS] 標記的最後一層隱藏狀態
query_emb = query_encoder(**query_input).last_hidden_state[:, 0, :]
ctx_emb = context_encoder(**ctx_input).last_hidden_state[:, 0, :]

# 使用點積計算相似度得分
score1 = query_emb @ ctx_emb[0]  # 341.3268
score2 = query_emb @ ctx_emb[1]  # 340.1626

高級用法

由於 Λ 從稀疏教師檢索器學習詞彙匹配，它可以與標準的密集檢索器（如 DPR、Contriever）結合使用，以構建在詞彙和語義匹配方面都表現出色的密集檢索器。

以下示例展示瞭如何通過連接DPR和Wiki BM25 Λ的嵌入來構建用於開放域問答的SPAR-Wiki模型：

import torch
from transformers import AutoTokenizer, AutoModel
from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer
from transformers import DPRContextEncoder, DPRContextEncoderTokenizer

# DPR模型
dpr_ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-multiset-base")
dpr_ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-multiset-base")
dpr_query_tokenizer = DPRQuestionEncoderTokenizer.from_pretrained("facebook/dpr-question_encoder-multiset-base")
dpr_query_encoder = DPRQuestionEncoder.from_pretrained("facebook/dpr-question_encoder-multiset-base")

# Wiki BM25 Λ模型
lexmodel_tokenizer = AutoTokenizer.from_pretrained('facebook/spar-wiki-bm25-lexmodel-query-encoder')
lexmodel_query_encoder = AutoModel.from_pretrained('facebook/spar-wiki-bm25-lexmodel-query-encoder')
lexmodel_context_encoder = AutoModel.from_pretrained('facebook/spar-wiki-bm25-lexmodel-context-encoder')

query = "Where was Marie Curie born?"
contexts = [
    "Maria Sklodowska, later known as Marie Curie, was born on November 7, 1867.",
    "Born in Paris on 15 May 1859, Pierre Curie was the son of Eugène Curie, a doctor of French Catholic origin from Alsace."
]

# 計算DPR嵌入
dpr_query_input = dpr_query_tokenizer(query, return_tensors='pt')['input_ids']
dpr_query_emb = dpr_query_encoder(dpr_query_input).pooler_output
dpr_ctx_input = dpr_ctx_tokenizer(contexts, padding=True, truncation=True, return_tensors='pt')
dpr_ctx_emb = dpr_ctx_encoder(**dpr_ctx_input).pooler_output

# 計算 Λ 嵌入
lexmodel_query_input = lexmodel_tokenizer(query, return_tensors='pt')
lexmodel_query_emb = lexmodel_query_encoder(**query_input).last_hidden_state[:, 0, :]
lexmodel_ctx_input = lexmodel_tokenizer(contexts, padding=True, truncation=True, return_tensors='pt')
lexmodel_ctx_emb = lexmodel_context_encoder(**ctx_input).last_hidden_state[:, 0, :]

# 通過連接形成SPAR嵌入

# 連接權重僅應用於查詢嵌入
# 詳情請參考SPAR論文
concat_weight = 0.7

spar_query_emb = torch.cat(
    [dpr_query_emb, concat_weight * lexmodel_query_emb],
    dim=-1,
    )
spar_ctx_emb = torch.cat(
    [dpr_ctx_emb, lexmodel_ctx_emb],
    dim=-1,
)

# 計算相似度得分
score1 = spar_query_emb @ spar_ctx_emb[0]  # 317.6931
score2 = spar_query_emb @ spar_ctx_emb[1]  # 314.6144