Update Summarization Bart Large Longformer

🚀 使用BART Large和Longformer编码器-解码器进行更新摘要生成

本项目是一个基于Transformer架构的模型，支持长文档的生成式序列到序列任务。它结合了BART Large和Longformer编码器 - 解码器，能够处理更长的输入，在更新摘要生成任务中表现出色。

🚀 快速开始

模型描述

此模型是一个基于Transformer的模型，支持长文档的生成式序列到序列任务。它基于 BART Large 并结合 Longformer编码器 - 解码器，以允许更长的输入。

预期用途与限制

如何使用

将数据进行格式化，使每个新文章或要添加的证据前面带有 <EV> 标记，每个标题前加上 <t> 前缀，每个摘要前加上 <abs> 前缀。请确保原始摘要也采用相同的格式。只要文章列表和原始摘要使用了正确的分隔标记，它们可以按任意顺序连接。

import torch
from transformers import LEDTokenizer, LEDForConditionalGeneration

tokenizer = LEDTokenizer.from_pretrained("hyesunyun/update-summarization-bart-large-longformer")
model = LEDForConditionalGeneration.from_pretrained("hyesunyun/update-summarization-bart-large-longformer")

input = "<EV> <t> Hypoglycemic effect of bitter melon compared with metformin in newly diagnosed type 2 diabetes patients. <abs> ETHNOPHARMACOLOGICAL RELEVANCE: Bitter melon (Momordica charantia L.) has been widely used as an traditional medicine treatment for diabetic patients in Asia. In vitro and animal studies suggested its hypoglycemic activity, but limited human studies are available to support its use. AIM OF STUDY: This study was conducted to assess the efficacy and safety of three doses of bitter melon compared with metformin. MATERIALS AND METHODS: This is a 4-week, multicenter, randomized, double-blind, active-control trial. Patients were randomized into 4 groups to receive bitter melon 500 mg/day, 1,000 mg/day, and 2,000 mg/day or metformin 1,000 mg/day. All patients were followed for 4 weeks. RESULTS: There was a significant decline in fructosamine at week 4 of the metformin group (-16.8; 95% CI, -31.2, -2.4 mumol/L) and the bitter melon 2,000 mg/day group (-10.2; 95% CI, -19.1, -1.3 mumol/L). Bitter melon 500 and 1,000 mg/day did not significantly decrease fructosamine levels (-3.5; 95% CI -11.7, 4.6 and -10.3; 95% CI -22.7, 2.2 mumol/L, respectively). CONCLUSIONS: Bitter melon had a modest hypoglycemic effect and significantly reduced fructosamine levels from baseline among patients with type 2 diabetes who received 2,000 mg/day. However, the hypoglycemic effect of bitter melon was less than metformin 1,000 mg/day. <EV> <t> Momordica charantia for type 2 diabetes mellitus. <abs> There is insufficient evidence to recommend momordica charantia for type 2 diabetes mellitus. Further studies are therefore required to address the issues of standardization and the quality control of preparations. For medical nutritional therapy, further observational trials evaluating the effects of momordica charantia are needed before RCTs are established to guide any recommendations in clinical practice."
inputs_dict = tokenizer(input, padding="max_length", max_length=10240, return_tensors="pt", truncation=True)
input_ids = inputs_dict.input_ids
attention_mask = inputs_dict.attention_mask
global_attention_mask = torch.zeros_like(attention_mask)
# put global attention on <s> token
global_attention_mask[:, 0] = 1

predicted_summary_ids = model.generate(input_ids, attention_mask=attention_mask, global_attention_mask=global_attention_mask)
print(tokenizer.batch_decode(predicted_summary_ids, skip_special_tokens=True))

限制和偏差

请提供潜在问题的示例以及可能的补救措施。

训练数据

使用预训练的 LED模型，并使用 此GitHub仓库 中的数据集进行微调。

训练过程

包括预处理、使用的硬件、超参数等。

评估结果

BibTeX引用和引用信息

@inproceedings{...,
  year={2021}
}

信息表格