Llama 3 2 3B Dpo Rlhf Fine Tuning

🚀 Fine-tuned Language Model for Preference Optimization (DPO)

This model is a fine - tuned version of Llama 3.2 - 3B - Instruct with Direct Preference Optimization (DPO). It's specialized for reward modeling tasks, offering high - performance in language comprehension, instruction - based response generation, and preference - based ranking of responses.

📚 Documentation

Model Overview

This model is a fine - tuned version of Llama 3.2 - 3B - Instruct with Direct Preference Optimization (DPO), specialized for reward modeling tasks. It has been optimized using memory - efficient techniques including 4 - bit quantization, gradient checkpointing, and parameter - efficient fine - tuning (PEFT). The model is tailored for tasks requiring language comprehension, instruction - based response generation, and preference - based ranking of responses.

Model Details

Model Capabilities

• Text Generation: Capable of generating detailed and coherent text responses based on instructions or prompts.
• Preference - Based Optimization: Fine - tuned to rank responses based on user feedback (chosen vs. rejected).
• Long Contexts: Supports processing up to 2048 tokens of input efficiently, facilitated by internal RoPE scaling.
• Faster Inference: Optimized for real - time text generation with streaming capabilities and low - latency responses.

Intended Use

This model can be applied to various natural language processing (NLP) tasks, including:

• Question Answering: Responding to user queries with detailed and contextually accurate information.
• Instruction Following: Generating responses based on user - defined tasks.
• Preference Modeling: Ranking different responses based on preferences provided in training data.
• Text Completion: Completing partially given texts based on provided instructions.

Limitations

• Context Length: While capable of processing up to 2048 tokens, extremely long texts may require additional optimization or truncation.
• Precision: The model's 4 - bit quantization may result in minor loss of precision in certain edge cases requiring high accuracy.
• Dataset Bias: Reflects biases present in the training dataset used for preference pairs labeling.

Technical Details

• Model Architecture: Based on Llama 3.2 with 3 billion parameters.
• Training Method: Fine - tuned using Direct Preference Optimization (DPO).
• Optimizer: Utilizes AdamW optimizer with 8 - bit precision for efficiency.
• Batch Size: Effective batch size of 8 (2 per device with 4 - step gradient accumulation).
• Training Configuration:
  • Learning rate: 5e - 6
  • Warm - up ratio: 0.1
  • Epochs: 1
  • Max sequence length: 2048 tokens
• Mixed Precision Training: Supports FP16 and BFloat16 depending on hardware.

Usage Instructions

📦 Installation

Ensure torch, transformers, unsloth, and other required libraries are installed for inference and fine - tuning.

💻 Usage Examples

Basic Usage

Load the model using FastLanguageModel.from_pretrained() by specifying the model name and optimization settings.

Advanced Usage

Apply PEFT and quantization strategies (e.g., LoRA, gradient checkpointing) using the dataset of preference pairs for fine - tuning. Use the FastLanguageModel.for_inference() method to enable optimized text generation, which supports streaming inference for real - time output.

Performance Metrics

• Training Loss: 1.19
• Training Runtime: 1974.06 seconds (approximately 32 minutes)
• Steps Per Second: 0.063
• Samples Per Second: 0.507

Model Version

• Version: Unsloth 2025.1.7 (Patched version)
• Training Date: January 2025

Acknowledgements

This model was trained using the Unsloth framework with contributions from Intel and Hugging Face for data and tools.

Notebook

Access the implementation notebook for this model here. This notebook provides detailed steps for fine - tuning and deploying the model.

📄 License

This model is released under the MIT license.