Llama_3_2_3B_SFT_GGUF Open-source AI Model - Pretrained and Fine-tuned for More Efficient Information Processing

Llama 3 2 3B SFT GGUF

Developed by SURESHBEEKHANI

Instruction-tuned version of the Llama-3.2-3B pre-trained model, utilizing 4-bit quantization and LoRA technology for efficient fine-tuning

Large Language Model EnglishOpen Source License:MIT #4-bit quantization #LoRA fine-tuning #QA task optimization

Downloads 53

Release Time : 1/21/2025

Model Overview

This model is a language model optimized for QA tasks, fine-tuned on the FineTome-100k dataset, featuring efficient inference capabilities and low VRAM consumption

Model Features

4-bit quantization

Significantly reduces VRAM usage, enabling the model to run on resource-limited devices

LoRA fine-tuning

Employs Low-Rank Adaptation for efficient fine-tuning, reducing training parameters while maintaining model performance

Efficient inference

Optimized inference speed suitable for real-time applications

Low VRAM usage

Peak VRAM consumption of only 3.855GB, ideal for resource-constrained environments

Model Capabilities

Text generation

Question answering systems

Instruction understanding and execution

Use Cases

Intelligent assistants

Knowledge QA system

Professional knowledge QA application built on the FineTome-100k dataset

Performs well in domain-specific QA tasks

Educational technology

Learning assistant tool

Intelligent tutoring system to help students with academic questions

🚀 Llama-3.2-3B-Instruct Fine-Tuning on Custom Dataset

This project demonstrates fine-tuning the Llama-3.2-3B-Instruct model on a custom dataset, using the Unsloth library with optimizations like 4-bit quantization and LoRA.

🚀 Quick Start

This repository shows how to fine-tune the Llama-3.2-3B-Instruct model with the Unsloth library. The model is trained on the custom FineTome-100k dataset for 60 steps. Key optimizations include:

4-bit quantization to cut down memory usage.
LoRA (Low-Rank Adaptation) for efficient fine-tuning.
Techniques to boost inference speed and generate text with the model.

✨ Features

Model Details

Property	Details
Model Name	Llama-3.2-3B-Instruct
Pretrained Weights	Unsloth’s pretrained version for Llama-3.2-3B
Quantization	4-bit quantization (set via `load_in_4bit=True`) for reduced memory usage

LoRA Configuration

Rank: 16
Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
LoRA Alpha: 16
LoRA Dropout: 0

Gradient Checkpointing

Use Gradient Checkpointing: "unsloth" for improved long-context training

Training

Dataset: FineTome-100k (first 500 records selected)
Loss Function: Standard loss for sequence-to-sequence tasks
Training Steps: 60 steps with batch size of 2 (gradient accumulation steps set to 4)
Optimizer: AdamW 8-bit

Training Parameters

Max Sequence Length: 2048 tokens
Learning Rate: 2e-4
Gradient Accumulation Steps: 4
Total Steps: 60
Epochs: 1 (as max_steps was set to 60)
Training Precision: Use FP16 or BF16 for training depending on GPU support

Performance

GPU Used: Tesla T4 (14.7 GB max memory)

Peak Memory Usage

Total Reserved Memory: 3.855 GB
Memory Used for LoRA: 1.312 GB
Memory Utilization: 26.1% (peak) of available memory

📚 Documentation

This notebook showcases an efficient approach to fine-tuning large language models with memory optimizations and improved training efficiency using LoRA and 4-bit quantization. The Unsloth library allows for fast training and inference, making this setup ideal for large-scale tasks even with limited GPU resources.

📄 License

This project is licensed under the MIT license.

Access the implementation notebook for this model here. This notebook provides detailed steps for fine-tuning and deploying the model.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご