Summllama3.1 8B

🚀 SummLlama3.1-8B

Are you in search of a summarizer capable of generating more human-preferred summaries across multiple domains? Our SummLlama3.1-8B might just be the solution you need! It is initialized from Llama3.1-8B-Instruct and undergoes additional training using Direct Preference Optimization (DPO) based on large-scale (over 100K) summarization feedback.

✨ Features

• Multi - domain Adaptability: The feedback used for training encompasses a wide variety of input documents, from short to long texts, including both dialogue and non - dialogue formats, spanning seven distinct domains: four non - dialogue domains (News, Lifestyle, Report, Medical) and three dialogue domains (Daily Life, Interview, Meeting).
• High - quality Summaries: This model excels at faithfulness, completeness, and conciseness, the three key aspects humans use to judge a good summarizer.
  • Faithfulness: It does not manipulate the information in the input text or add any information not directly inferable from it.
  • Completeness: Ensures all key information from the input text is included in the output summary.
  • Conciseness: Refrains from incorporating non - key information in the output, maintaining a succinct and focused summary.

📚 Documentation

Automated Evaluation Results

Human Evaluation Results

Automated Evaluation using FineSurE

Model Links

• SummLlama3 - 70B
  • https://huggingface.co/DISLab/SummLlama3 - 8B
  • https://huggingface.co/DISLab/SummLlama3 - 70B
• SummLlama3.1 - Series
  • https://huggingface.co/DISLab/SummLlama3.1 - 8B
  • https://huggingface.co/DISLab/SummLlama3.1 - 70B
• SummLlama3.2 - Series
  • https://huggingface.co/DISLab/SummLlama3.2 - 3B

Recommended Prompt for Text Summarization

We recommend using the following prompt to get the summary, as the model was trained with it.

def format_chat_template(document):
    instruction = "Please summarize the input documnet."
    row_json = [{"role": "user", "content": f"Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Input:\n{document}\n\n### Response:\n"}]
    return tokenizer.apply_chat_template(row_json, tokenize=False)

💻 Usage Examples

Example Dialogue

Summary Comparison

Advantages of SummLlama3 - 8B Summary

The summary of SummLlama3 - 8B can be considered a much human - preferred summary for the following reasons:

• Core Focus: The summary accurately captures the main theme of the conversation, which revolves around the Thanksgiving dinner arrangements. It highlights how the two people confirm plans, discuss what to bring, and finalize the decision for Person2 to bring wine instead of pie. This maintains the core context.
• Inclusion of Key - facts: The summary covers the important details of the conversation, including Person2's initial offer to bring dessert (pumpkin pie) and the shift to bringing wine due to another family member handling dessert. Other summaries tend to overlook or simplify this progression, while SummLlama3 - 8B fully captures the interaction’s key events.
• Clarity and Conciseness: The summary is structured in a straightforward, concise manner, effectively summarizing the conversation without unnecessary details. It presents the flow and outcome of the discussion clearly, making it easy for readers to understand. The logical order of events is maintained, ensuring a smooth narrative.
• Accurate Role Depiction: The summary clearly identifies Person1 as the host and Paul (Person2) as the guest, which helps clarify their relationship and the nature of the conversation. This distinction is more explicit in SummLlama3 - 8B compared to other summaries, which might leave these roles more ambiguous.

Please refer to our paper to learn how to exploit LLM - generated feedback in the context of text summarization.