Mbart Ru Sum Gazeta

🚀 MBARTRuSumGazeta

這是一個用於俄語新聞自動摘要的模型，基於MBART架構，在特定數據集上訓練，能有效生成新聞文本的摘要，提升信息獲取效率。

🚀 快速開始

你可以通過Colab快速體驗該模型：鏈接

from transformers import MBartTokenizer, MBartForConditionalGeneration

model_name = "IlyaGusev/mbart_ru_sum_gazeta"
tokenizer = MBartTokenizer.from_pretrained(model_name)
model = MBartForConditionalGeneration.from_pretrained(model_name)

article_text = "..."

input_ids = tokenizer(
    [article_text],
    max_length=600,
    padding="max_length",
    truncation=True,
    return_tensors="pt",
)["input_ids"]

output_ids = model.generate(
    input_ids=input_ids,
    no_repeat_ngram_size=4
)[0]

summary = tokenizer.decode(output_ids, skip_special_tokens=True)
print(summary)

✨ 主要特性

• 這是fairseq模型的移植版本。
• 更多詳細信息請參考俄語新聞自動摘要數據集。

📦 安裝指南

文檔未提供具體安裝步驟，可參考Colab鏈接中的環境配置。

💻 使用示例

基礎用法

from transformers import MBartTokenizer, MBartForConditionalGeneration

model_name = "IlyaGusev/mbart_ru_sum_gazeta"
tokenizer = MBartTokenizer.from_pretrained(model_name)
model = MBartForConditionalGeneration.from_pretrained(model_name)

article_text = "..."

input_ids = tokenizer(
    [article_text],
    max_length=600,
    padding="max_length",
    truncation=True,
    return_tensors="pt",
)["input_ids"]

output_ids = model.generate(
    input_ids=input_ids,
    no_repeat_ngram_size=4
)[0]

summary = tokenizer.decode(output_ids, skip_special_tokens=True)
print(summary)

高級用法

預測所有摘要：

import json
import torch
from transformers import MBartTokenizer, MBartForConditionalGeneration
from datasets import load_dataset


def gen_batch(inputs, batch_size):
    batch_start = 0
    while batch_start < len(inputs):
        yield inputs[batch_start: batch_start + batch_size]
        batch_start += batch_size


def predict(
    model_name,
    input_records,
    output_file,
    max_source_tokens_count=600,
    batch_size=4
):
    device = "cuda" if torch.cuda.is_available() else "cpu"
    
    tokenizer = MBartTokenizer.from_pretrained(model_name)
    model = MBartForConditionalGeneration.from_pretrained(model_name).to(device)
    
    predictions = []
    for batch in gen_batch(inputs, batch_size):
        texts = [r["text"] for r in batch]
        input_ids = tokenizer(
            batch,
            return_tensors="pt",
            padding="max_length",
            truncation=True,
            max_length=max_source_tokens_count
        )["input_ids"].to(device)
        
        output_ids = model.generate(
            input_ids=input_ids,
            no_repeat_ngram_size=4
        )
        summaries = tokenizer.batch_decode(output_ids, skip_special_tokens=True)
        for s in summaries:
            print(s)
        predictions.extend(summaries)
    with open(output_file, "w") as w:
        for p in predictions:
            w.write(p.strip().replace("\n", " ") + "\n")

gazeta_test = load_dataset('IlyaGusev/gazeta', script_version="v1.0")["test"]
predict("IlyaGusev/mbart_ru_sum_gazeta", list(gazeta_test), "mbart_predictions.txt")

評估：https://github.com/IlyaGusev/summarus/blob/master/evaluate.py 標誌：--language ru --tokenize-after --lower

📚 詳細文檔

預期用途與限制

如何使用

可通過Colab鏈接快速體驗模型。

限制和偏差

• 該模型在Gazeta.ru的文章上表現良好，但對於其他機構的文章可能會受到領域偏移的影響。

訓練數據

• 數據集：Gazeta

訓練過程

• Fairseq訓練腳本：train.sh
• 移植：Colab鏈接

評估結果

Gazeta v1 數據集

Gazeta v2 數據集

🔧 技術細節

文檔未提供具體技術細節。

📄 許可證

該項目使用的許可證為：apache - 2.0

BibTeX引用和引用信息

@InProceedings{10.1007/978-3-030-59082-6_9,
    author="Gusev, Ilya",
    editor="Filchenkov, Andrey and Kauttonen, Janne and Pivovarova, Lidia",
    title="Dataset for Automatic Summarization of Russian News",
    booktitle="Artificial Intelligence and Natural Language",
    year="2020",
    publisher="Springer International Publishing",
    address="Cham",
    pages="122--134",
    isbn="978-3-030-59082-6"
}