Re2g Qry Encoder Fever

🚀 Re2G中FEVER問題編碼器的模型卡片

本模型用於將問題編碼為向量，可作為近似最近鄰索引的查詢。它結合上下文編碼器，能在信息檢索等任務中發揮重要作用。

📚 詳細文檔

模型詳情

RAG、Multi - DPR和KGI的方法是訓練一個神經信息檢索（IR）組件，並通過其在生成正確輸出中的影響進行端到端訓練。

訓練、評估和推理

訓練、評估和推理的代碼可在我們的GitHub倉庫的re2g分支中找到。

使用方法

使用該模型的最佳方式是調整dpr_apply.py。

引用信息

@inproceedings{glass-etal-2022-re2g,
    title = "{R}e2{G}: Retrieve, Rerank, Generate",
    author = "Glass, Michael  and
      Rossiello, Gaetano  and
      Chowdhury, Md Faisal Mahbub  and
      Naik, Ankita  and
      Cai, Pengshan  and
      Gliozzo, Alfio",
    booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    month = jul,
    year = "2022",
    address = "Seattle, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.naacl-main.194",
    doi = "10.18653/v1/2022.naacl-main.194",
    pages = "2701--2715",
    abstract = "As demonstrated by GPT-3 and T5, transformers grow in capability as parameter spaces become larger and larger. However, for tasks that require a large amount of knowledge, non-parametric memory allows models to grow dramatically with a sub-linear increase in computational cost and GPU memory requirements. Recent models such as RAG and REALM have introduced retrieval into conditional generation. These models incorporate neural initial retrieval from a corpus of passages. We build on this line of research, proposing Re2G, which combines both neural initial retrieval and reranking into a BART-based sequence-to-sequence generation. Our reranking approach also permits merging retrieval results from sources with incomparable scores, enabling an ensemble of BM25 and neural initial retrieval. To train our system end-to-end, we introduce a novel variation of knowledge distillation to train the initial retrieval, reranker and generation using only ground truth on the target sequence output. We find large gains in four diverse tasks: zero-shot slot filling, question answering, fact checking and dialog, with relative gains of 9{\%} to 34{\%} over the previous state-of-the-art on the KILT leaderboard. We make our code available as open source.",
}

模型描述

模型創建者在相關論文中指出：

正如GPT - 3和T5所展示的，隨著參數空間越來越大，Transformer的能力也在不斷增強。然而，對於需要大量知識的任務，非參數內存允許模型以亞線性的計算成本和GPU內存需求大幅增長。最近的模型如RAG和REALM已將檢索引入到條件生成中。這些模型包含了從段落語料庫進行的神經初始檢索。我們在此研究基礎上，提出了Re2G，它將神經初始檢索和重排序結合到基於BART的序列到序列生成中。我們的重排序方法還允許合併來自分數不可比來源的檢索結果，從而實現BM25和神經初始檢索的集成。為了端到端地訓練我們的系統，我們引入了一種新穎的知識蒸餾變體，僅使用目標序列輸出的真實值來訓練初始檢索、重排序器和生成器。我們在四個不同的任務中取得了顯著的收益：零樣本插槽填充、問答、事實核查和對話，在KILT排行榜上相對於先前的最先進技術有9%到34%的相對提升。我們將代碼開源。

📖 使用場景

直接使用

該模型可用於將問題編碼為向量，作為近似最近鄰索引的查詢。它必須與將段落編碼為向量並進行索引的上下文編碼器結合使用。

📄 許可證

本模型採用Apache 2.0許可證。

📜 引用

BibTeX格式：

@inproceedings{glass-etal-2022-re2g,
    title = "{R}e2{G}: Retrieve, Rerank, Generate",
    author = "Glass, Michael  and
      Rossiello, Gaetano  and
      Chowdhury, Md Faisal Mahbub  and
      Naik, Ankita  and
      Cai, Pengshan  and
      Gliozzo, Alfio",
    booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    month = jul,
    year = "2022",
    address = "Seattle, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.naacl-main.194",
    doi = "10.18653/v1/2022.naacl-main.194",
    pages = "2701--2715",
    abstract = "As demonstrated by GPT-3 and T5, transformers grow in capability as parameter spaces become larger and larger. However, for tasks that require a large amount of knowledge, non-parametric memory allows models to grow dramatically with a sub-linear increase in computational cost and GPU memory requirements. Recent models such as RAG and REALM have introduced retrieval into conditional generation. These models incorporate neural initial retrieval from a corpus of passages. We build on this line of research, proposing Re2G, which combines both neural initial retrieval and reranking into a BART-based sequence-to-sequence generation. Our reranking approach also permits merging retrieval results from sources with incomparable scores, enabling an ensemble of BM25 and neural initial retrieval. To train our system end-to-end, we introduce a novel variation of knowledge distillation to train the initial retrieval, reranker and generation using only ground truth on the target sequence output. We find large gains in four diverse tasks: zero-shot slot filling, question answering, fact checking and dialog, with relative gains of 9{\%} to 34{\%} over the previous state-of-the-art on the KILT leaderboard. We make our code available as open source.",
}