Llama 3.1 8B Tldr

🚀 Llama-3.1-8B-tldr

This is a fine - tuned version of Llama-3.1-8B, designed to summarize text in the style of Reddit posts, achieving a BERTScore of 0.366 on the test set of trl - lib/tldr.

🚀 Quick Start

This model is a fine - tuned version of [meta - llama/Llama - 3.1 - 8B](https://huggingface.co/meta - llama/Llama - 3.1 - 8B) on the [trl - lib/tldr](https://huggingface.co/datasets/trl - lib/tldr) dataset. It can be used for text summarization in the style of Reddit posts.

✨ Features

• Model Architecture: LlamaForCausalLM, taking text as input and outputting text.
• Release Date: 05/29/2025
• Version: 1.0
• Intended Use Cases: Finetuned to summarize text in the style of Reddit posts.
• Out - of - scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws) and any other way prohibited by the Acceptable Use Policy and Llama 3.1 Community License.
• Model Developers: Red Hat (Neural Magic)
• Performance: Obtains 0.366 BERTScore on the test set of trl - lib/tldr.

📦 Installation

No specific installation steps are provided in the original document. However, for deployment, you can use vLLM.

💻 Usage Examples

Basic Usage

Run the following command to start the vLLM server:

vllm serve RedHatAI/Llama-3.1-8B-tldr

Advanced Usage

Once your server is started, you can query the model using the OpenAI API:

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

post="""
SUBREDDIT: r/AI

TITLE: Training sparse LLMs

POST: Now you can use the llm-compressor integration to axolotl to train sparse LLMs!

It's super easy to use. See the example in https://huggingface.co/RedHatAI/Sparse-Llama-3.1-8B-tldr-2of4.

And there's more. You can run 2:4 sparse models on vLLM and get significant speedupts on Hopper GPUs!
"""

prompt = f"Give a TL;DR of the following Reddit post.\n<|user|>{post}\nTL;DR:\n<|assistant|>\n"

completion = client.completions.create(
  model="RedHatAI/Llama-3.1-8B-tldr",
  prompt=prompt,
  max_tokens=256,
)
print("Completion result:", completion)

📚 Documentation

Training

base_model: meta-llama/Llama-3.1-8B

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: trl-lib/tldr
    type:
      system_prompt: "Give a TL;DR of the following Reddit post."
      field_system: system
      field_instruction: prompt
      field_output: completion
      format: "<|user|>\n{instruction}\n<|assistant|>\n"
      no_input_format: "<|user|>\n{instruction}\n<|assistant|>\n"
    split: train

sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true
eval_sample_packing: true

torch.compile: true
gradient_accumulation_steps: 1
micro_batch_size: 4
num_epochs: 3
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 1e-5
max_grad_norm: 1

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false

train_on_inputs: false
bf16: auto
fp16:
tf32: false

early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
flash_attention: true

warmup_ratio: 0.05
evals_per_epoch: 4
val_set_size: 0.05
save_strategy: "best"
save_total_limit: 1
metric_for_best_model: "loss"

debug:
deepspeed:
weight_decay: 0.0
special_tokens:
  pad_token: "<|end_of_text|>"

seed: 0

plugins:
  - axolotl.integrations.liger.LigerPlugin

liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true
liger_fused_linear_cross_entropy: true

Evaluation

The model was evaluated on the test split of [trl - lib/tldr](https://huggingface.co/datasets/trl - lib/tldr) using the Neural Magic fork of [lm - evaluation - harness](https://github.com/neuralmagic/lm - evaluation - harness/tree/tldr) (tldr branch). One can reproduce these results by using the following command:

lm_eval --model vllm --model_args "pretrained=RedHatAI/Llama-3.1-8B-tldr,dtype=auto,add_bos_token=True" --batch-size auto --tasks tldr

Inference Performance

We evaluated the inference performance of this model using the first 1,000 samples from the training set of the [trl - lib/tldr](https://huggingface.co/datasets/trl - lib/tldr) dataset. Benchmarking was conducted with vLLM version 0.9.0.1 and GuideLLM version 0.2.1.

The figure below presents the mean end - to - end latency per request across varying request rates. Results are shown for this model, as well as two variants:

• Dense - quantized: [Llama - 3.1 - 8B - tldr - FP8 - dynamic](https://huggingface.co/RedHatAI/Llama - 3.1 - 8B - tldr - FP8 - dynamic)
• Sparse - quantized: [Sparse - Llama - 3.1 - 8B - tldr - 2of4 - FP8 - dynamic](https://huggingface.co/RedHatAI/Sparse - Llama - 3.1 - 8B - tldr - 2of4 - FP8 - dynamic)

from datasets import load_dataset
ds = load_dataset("trl-lib/tldr", split="train").take(1000)
ds.to_json("tldr_1000.json")

vllm serve RedHatAI/Llama-3.1-8B-tldr

GUIDELLM__OPENAI__MAX_OUTPUT_TOKENS=128 guidellm benchmark --target "http://localhost:8000" --rate-type sweep --data tldr_1000.json

📄 License

The license of this model is llama3.1.