Update Summarization Bart Large Longformer

🚀 使用BART Large和Longformer編碼器-解碼器進行更新摘要生成

本項目是一個基於Transformer架構的模型，支持長文檔的生成式序列到序列任務。它結合了BART Large和Longformer編碼器 - 解碼器，能夠處理更長的輸入，在更新摘要生成任務中表現出色。

🚀 快速開始

模型描述

此模型是一個基於Transformer的模型，支持長文檔的生成式序列到序列任務。它基於 BART Large 並結合 Longformer編碼器 - 解碼器，以允許更長的輸入。

預期用途與限制

如何使用

將數據進行格式化，使每個新文章或要添加的證據前面帶有 <EV> 標記，每個標題前加上 <t> 前綴，每個摘要前加上 <abs> 前綴。請確保原始摘要也採用相同的格式。只要文章列表和原始摘要使用了正確的分隔標記，它們可以按任意順序連接。

import torch
from transformers import LEDTokenizer, LEDForConditionalGeneration

tokenizer = LEDTokenizer.from_pretrained("hyesunyun/update-summarization-bart-large-longformer")
model = LEDForConditionalGeneration.from_pretrained("hyesunyun/update-summarization-bart-large-longformer")

input = "<EV> <t> Hypoglycemic effect of bitter melon compared with metformin in newly diagnosed type 2 diabetes patients. <abs> ETHNOPHARMACOLOGICAL RELEVANCE: Bitter melon (Momordica charantia L.) has been widely used as an traditional medicine treatment for diabetic patients in Asia. In vitro and animal studies suggested its hypoglycemic activity, but limited human studies are available to support its use. AIM OF STUDY: This study was conducted to assess the efficacy and safety of three doses of bitter melon compared with metformin. MATERIALS AND METHODS: This is a 4-week, multicenter, randomized, double-blind, active-control trial. Patients were randomized into 4 groups to receive bitter melon 500 mg/day, 1,000 mg/day, and 2,000 mg/day or metformin 1,000 mg/day. All patients were followed for 4 weeks. RESULTS: There was a significant decline in fructosamine at week 4 of the metformin group (-16.8; 95% CI, -31.2, -2.4 mumol/L) and the bitter melon 2,000 mg/day group (-10.2; 95% CI, -19.1, -1.3 mumol/L). Bitter melon 500 and 1,000 mg/day did not significantly decrease fructosamine levels (-3.5; 95% CI -11.7, 4.6 and -10.3; 95% CI -22.7, 2.2 mumol/L, respectively). CONCLUSIONS: Bitter melon had a modest hypoglycemic effect and significantly reduced fructosamine levels from baseline among patients with type 2 diabetes who received 2,000 mg/day. However, the hypoglycemic effect of bitter melon was less than metformin 1,000 mg/day. <EV> <t> Momordica charantia for type 2 diabetes mellitus. <abs> There is insufficient evidence to recommend momordica charantia for type 2 diabetes mellitus. Further studies are therefore required to address the issues of standardization and the quality control of preparations. For medical nutritional therapy, further observational trials evaluating the effects of momordica charantia are needed before RCTs are established to guide any recommendations in clinical practice."
inputs_dict = tokenizer(input, padding="max_length", max_length=10240, return_tensors="pt", truncation=True)
input_ids = inputs_dict.input_ids
attention_mask = inputs_dict.attention_mask
global_attention_mask = torch.zeros_like(attention_mask)
# put global attention on <s> token
global_attention_mask[:, 0] = 1

predicted_summary_ids = model.generate(input_ids, attention_mask=attention_mask, global_attention_mask=global_attention_mask)
print(tokenizer.batch_decode(predicted_summary_ids, skip_special_tokens=True))

限制和偏差

請提供潛在問題的示例以及可能的補救措施。

訓練數據

使用預訓練的 LED模型，並使用 此GitHub倉庫 中的數據集進行微調。

訓練過程

包括預處理、使用的硬件、超參數等。

評估結果

BibTeX引用和引用信息

@inproceedings{...,
  year={2021}
}

信息表格