Finetuned T5 Xsum

🚀 LoRA微調的XSum T5文本摘要器

本模型是一個專為文本摘要任務優化的工具，基於t5-small模型在XSum數據集上進行LoRA（低秩自適應）微調，能高效生成高質量的文本摘要。

🚀 快速開始

本模型是 t5-small 在xsum數據集上的微調版本。

✨ 主要特性

這是一個針對文本摘要進行優化的T5-small的LoRA（低秩自適應）微調版本。該模型在XSum數據集上進行了抽象摘要的訓練。

💻 使用示例

基礎用法

from peft import PeftModel
from transformers import AutoModelForSeq2SeqLM
from transformers import AutoTokenizer
import torch

base_model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
my_model = PeftModel.from_pretrained(base_model, "Lakshan2003/finetuned-t5-xsum")

def test_peft_summarizer(text, model, max_length=128, min_length=30):
    """
    Test the PEFT-loaded summarization model
    
    Args:
        text (str): Input text to summarize
        model: The loaded PEFT model
        max_length (int): Maximum length of the summary
        min_length (int): Minimum length of the summary
    """
    # Load tokenizer for t5-small (base model)
    tokenizer = AutoTokenizer.from_pretrained("Lakshan2003/finetuned-t5-xsum")
    
    # Move model to GPU if available
    device = "cuda" if torch.cuda.is_available() else "cpu"
    model = model.to(device)
    
    # Prepare the input text
    prefix = "summarize: "
    input_text = prefix + text
    
    # Tokenize
    inputs = tokenizer(input_text, return_tensors="pt", max_length=512, truncation=True)
    inputs = {k: v.to(device) for k, v in inputs.items()}
    
    # Generate summary
    with torch.no_grad():
        output_ids = model.generate(
            input_ids=inputs["input_ids"],
            attention_mask=inputs["attention_mask"],
            max_length=max_length,
            min_length=min_length,
            num_beams=4,
            length_penalty=2.0,
            early_stopping=True,
            no_repeat_ngram_size=3
        )
    
    # Decode the summary
    summary = tokenizer.decode(output_ids[0], skip_special_tokens=True)
    
    return summary

# Test text
test_text = """
The United Nations has warned that climate change poses an unprecedented threat to human civilization. In a landmark report, scientists detailed how rising temperatures are affecting everything from weather patterns to food production. The report emphasizes that without immediate and substantial action to reduce greenhouse gas emissions, the world faces severe consequences including rising sea levels, more frequent extreme weather events, and widespread ecosystem collapse. Many countries have pledged to reduce their carbon emissions, but experts say current commitments fall short of what's needed to prevent the worst impacts of climate change. The report also highlights the disproportionate effect of climate change on developing nations, which often lack the resources to adapt to changing conditions.
"""

# Generate summary
summary = test_peft_summarizer(test_text, my_model)

print("Original Text:")
print(test_text)
print("\nGenerated Summary:")
print(summary)

🔧 技術細節

訓練超參數

訓練過程中使用了以下超參數：

• 學習率：2e-05
• 訓練批次大小：4
• 評估批次大小：4
• 隨機種子：42
• 梯度累積步數：2
• 總訓練批次大小：8
• 優化器：使用adamw_torch，β值為(0.9, 0.999)，ε值為1e-08，無額外優化器參數
• 學習率調度器類型：線性
• 訓練輪數：1
• 混合精度訓練：原生自動混合精度（Native AMP）

框架版本

• PEFT 0.14.0
• Transformers 4.47.0
• Pytorch 2.5.1+cu121
• Datasets 3.2.0
• Tokenizers 0.21.0

📄 許可證

本項目採用Apache-2.0許可證。