Summllama3.2 3B

🚀 SummLlama3.2-3B

A powerful summarizer that generates human-preferred summaries across multiple domains.

Are you in search of a summarizer capable of generating more human-preferred summaries across various domains? Our SummLlama3.2-3B might be the exact solution you need!

SummLlama3.2-3B is initialized from Llama3.2-3B-Instruct and undergoes additional training using Direct Preference Optimization (DPO) based on large-scale (over 100K) summarization feedback. This feedback encompasses a wide array of input documents, ranging from short to long texts, including both dialogue and non-dialogue formats, and spanning seven distinct domains:

• Four non-dialogue domains: News, Lifestyle, Report, Medical
• Three dialogue domains: Daily Life, Interview, Meeting

✨ Features

Automated Evaluation Results

Please refer to our paper to understand how to exploit LLM-generated feedback in the context of text summarization.

Related Models

• SummLlama3-70B
  • https://huggingface.co/DISLab/SummLlama3-8B
  • https://huggingface.co/DISLab/SummLlama3-70B
• SummLlama3.1-Series
  • https://huggingface.co/DISLab/SummLlama3.1-8B
  • https://huggingface.co/DISLab/SummLlama3.1-70B
• SummLlama3.2-Series
  • https://huggingface.co/DISLab/SummLlama3.2-3B

Recommended Prompt for Text Summarization

We recommend using the following prompt to obtain summaries, as the model was trained with it.

def format_chat_template(document):
    instruction = "Please summarize the input document."
    row_json = [{"role": "user", "content": f"Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Input:\n{document}\n\n### Response:\n"}]
    return tokenizer.apply_chat_template(row_json, tokenize=False)

🔧 Technical Details

Rather than relying on expensive human feedback, we utilize high-quality, multi-dimensional, and fine-grained feedback generated by large language models (LLMs). This model excels at faithfulness, completeness, and conciseness, which are the three human-preferred aspects for judging a good summarizer:

• Faithfulness: A summarizer does not manipulate the information in the input text and does not add any information not directly inferable from the input text.
• Completeness: A summarizer ensures the inclusion of all key information from the input text in the output summary.
• Conciseness: A summarizer refrains from incorporating information outside the key information in the output, maintaining a succinct and focused summary.

Based on our comprehensive evaluation, which included both human and automated assessments of summary quality, SummLlama3 demonstrated significant improvements over the original Llama3 series.

Human Evaluation

Automated Evaluation using FineSurE

💻 Usage Examples

Example

See an example of how the summary improved by SummLlama3-8B over Llama3-8/70B-Instruct on the following document:

The summary of SummLlama3-8B can be considered a much human-preferred summary for the following reasons:

• Core Focus: The summary accurately captures the main theme of the conversation, which revolves around the Thanksgiving dinner arrangements. It highlights how the two people confirm plans, discuss what to bring, and finalize the decision for Person2 to bring wine instead of pie. This maintains the core context.
• Inclusion of Key-facts: The summary covers the important details of the conversation, including Person2's initial offer to bring dessert (pumpkin pie) and the shift to bringing wine due to another family member handling dessert. Other summaries tend to overlook or simplify this progression, while SummLlama3-8B fully captures the interaction’s key events.
• Clarity and Conciseness: The summary is structured in a straightforward, concise manner, effectively summarizing the conversation without unnecessary details. It presents the flow and outcome of the discussion clearly, making it easy for readers to understand. The logical order of events is maintained, ensuring a smooth narrative.
• Accurate Role Depiction: The summary clearly identifies Person1 as the host and Paul (Person2) as the guest, which helps clarify their relationship and the nature of the conversation. This distinction is more explicit in SummLlama3-8B compared to other summaries, which might leave these roles more ambiguous.