Keybart

🚀 KeyBART

KeyBART 是在《Learning Rich Representations of Keyphrase from Text》（发表于 2022 年北美计算语言学协会会议（NAACL）成果集，论文链接：https://aclanthology.org/2022.findings-naacl.67.pdf）中提出的模型。它基于 BART 架构进行预训练，以 CatSeqD 格式生成关键词组的拼接序列。

我们提供了一些下游评估设置的示例，以及如何在零样本设置下将其用于文本到文本生成任务。

🚀 快速开始

你可以参考以下不同任务的代码示例来快速使用 KeyBART。

✨ 主要特性

• 基于 BART 架构预训练，能够生成关键词组的拼接序列。
• 在关键词生成、抽象摘要等下游任务中表现良好。
• 支持零样本设置下的文本到文本生成。

📦 安装指南

文档未提及具体安装步骤，可通过以下代码加载相关模型和数据集：

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from datasets import load_dataset

# 加载 KeyBART 模型和分词器
tokenizer = AutoTokenizer.from_pretrained("bloomberg/KeyBART")
model = AutoModelForSeq2SeqLM.from_pretrained("bloomberg/KeyBART")

# 加载数据集示例
dataset = load_dataset("midas/kp20k")

💻 使用示例

基础用法

关键词生成

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from datasets import load_dataset

# 加载 KeyBART 模型和分词器
tokenizer = AutoTokenizer.from_pretrained("bloomberg/KeyBART")
model = AutoModelForSeq2SeqLM.from_pretrained("bloomberg/KeyBART")

# 加载数据集
dataset = load_dataset("midas/kp20k")

抽象摘要

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from datasets import load_dataset

# 加载 KeyBART 模型和分词器
tokenizer = AutoTokenizer.from_pretrained("bloomberg/KeyBART")
model = AutoModelForSeq2SeqLM.from_pretrained("bloomberg/KeyBART")

# 加载数据集
dataset = load_dataset("cnn_dailymail")

零样本设置

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

# 加载 KeyBART 模型和分词器
tokenizer = AutoTokenizer.from_pretrained("bloomberg/KeyBART")
model = AutoModelForSeq2SeqLM.from_pretrained("bloomberg/KeyBART")

或者使用 https://huggingface.co/bloomberg/KeyBART 提供的托管推理 API 控制台。

高级用法

零样本设置示例

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

# 加载 KeyBART 模型和分词器
tokenizer = AutoTokenizer.from_pretrained("bloomberg/KeyBART")
model = AutoModelForSeq2SeqLM.from_pretrained("bloomberg/KeyBART")

# 输入文本
input_text = "In this work, we explore how to learn task specific language models aimed towards learning rich representation of keyphrases from text documents. \
We experiment with different masking strategies for pre-training transformer language models (LMs) in discriminative as well as generative settings. \
In the discriminative setting, we introduce a new pre-training objective - Keyphrase Boundary Infilling with Replacement (KBIR), \
showing large gains in performance (upto 9.26 points in F1) over SOTA, when LM pre-trained using KBIR is fine-tuned for the task of keyphrase extraction. \
In the generative setting, we introduce a new pre-training setup for BART - KeyBART, that reproduces the keyphrases related to the input text in the CatSeq \
format, instead of the denoised original input. This also led to gains in performance (upto 4.33 points in F1@M) over SOTA for keyphrase generation. \
Additionally, we also fine-tune the pre-trained language models on named entity recognition (NER), question answering (QA), relation extraction (RE), \
abstractive summarization and achieve comparable performance with that of the SOTA, showing that learning rich representation of keyphrases is indeed beneficial \
for many other fundamental NLP tasks."

# 输出结果示例
print("Output: language model;keyphrase generation;new pre-training objective;pre-training setup;")

📚 详细文档

下游评估

关键词生成

以下是不同模型在关键词生成任务上的报告结果：

现有关键词生成

缺失关键词生成

抽象摘要

以下是不同模型在抽象摘要任务上的报告结果：

零样本设置

可通过以下代码进行零样本设置：

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("bloomberg/KeyBART")
model = AutoModelForSeq2SeqLM.from_pretrained("bloomberg/KeyBART")

或者使用 https://huggingface.co/bloomberg/KeyBART 提供的托管推理 API 控制台。

📄 许可证

本项目采用 Apache-2.0 许可证。

🔗 引用

请使用以下 BibTeX 条目引用此工作：

@inproceedings{kulkarni-etal-2022-learning,
    title = "Learning Rich Representation of Keyphrases from Text",
    author = "Kulkarni, Mayank  and
      Mahata, Debanjan  and
      Arora, Ravneet  and
      Bhowmik, Rajarshi",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2022",
    month = jul,
    year = "2022",
    address = "Seattle, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.findings-naacl.67",
    doi = "10.18653/v1/2022.findings-naacl.67",
    pages = "891--906",
    abstract = "In this work, we explore how to train task-specific language models aimed towards learning rich representation of keyphrases from text documents. We experiment with different masking strategies for pre-training transformer language models (LMs) in discriminative as well as generative settings. In the discriminative setting, we introduce a new pre-training objective - Keyphrase Boundary Infilling with Replacement (KBIR), showing large gains in performance (upto 8.16 points in F1) over SOTA, when the LM pre-trained using KBIR is fine-tuned for the task of keyphrase extraction. In the generative setting, we introduce a new pre-training setup for BART - KeyBART, that reproduces the keyphrases related to the input text in the CatSeq format, instead of the denoised original input. This also led to gains in performance (upto 4.33 points in F1@M) over SOTA for keyphrase generation. Additionally, we also fine-tune the pre-trained language models on named entity recognition (NER), question answering (QA), relation extraction (RE), abstractive summarization and achieve comparable performance with that of the SOTA, showing that learning rich representation of keyphrases is indeed beneficial for many other fundamental NLP tasks.",
}

如有任何问题，请联系 dmahata@bloomberg.net。