Qwen 2.5 7B DTF

🚀 Continued Pretrained Qwen2.5-7B for DTF Posts

This is a continued pretrained version of the unsloth/Qwen2.5-7B model. It uses unsloth's low rank adaptation on a dataset of DTF posts, and the adapter is already merged with the model.

🚀 Quick Start

This model is a continued pretrained version of the unsloth/Qwen2.5-7B model, leveraging unsloth's low rank adaptation on a dataset of DTF posts. The adapter has already been merged with the model.

📦 Installation

No specific installation steps provided in the original document.

💻 Usage Examples

No code examples provided in the original document.

📚 Documentation

Dataset Selection

For pretraining, posts from SubMaroon/DTF_comments_Responses_Counts were selected. They were deduplicated using the simple df.unique method and filtered by a token length between 1000 and 128000. The training dataset size was approximately 75M tokens.

LoRA Hyperparameters

r=32
target_modules=[
    "q_proj", 
    "k_proj", 
    "v_proj", 
    "o_proj",
    "gate_proj",
    "up_proj",
    "down_proj",
]
lora_alpha=16
lora_dropout=0
bias="none"
use_gradient_checkpointing='unsloth'
use_rslora=True
random_state=42

Training Hyperparameters

num_train_epochs=2
train_batch_size=8
gradient_accumulation_steps=16
gradient_checkpointing=False
optim="adamw_8bit"
weight_decay=4e-2
bf16=True
learning_rate=5e-5
lr_scheduler_type="cosine"
packing=True,
seed=42

Training Time

• NVidia Tesla A100 80GB: ~8.5 hours
• NVidia RTX 3090ti: ~33.5 hours

Related Links

• Wandb
• GitHub: TODO

🔧 Technical Details

The model is a continued pretrained version of unsloth/Qwen2.5-7B. It uses low rank adaptation (LoRA) on a specific DTF post dataset. The dataset was pre - processed by deduplication and length filtering. The LoRA and training hyperparameters are carefully set to achieve better performance.

📄 License

This project is licensed under the Apache 2.0 license.