Llama 3.3 70B Vulpecula R1

Developed by Sao10K

A thought-oriented model fine-tuned based on Meta's Llama 3.3-70B-Instruct, focusing on enhancing creative writing and role-playing capabilities

Large Language Model

Transformers

English#Chain of Thought Activation #Creative Writing Enhancement #Role-playing Optimization

Downloads 1,279

Release Time : 3/20/2025

Model Overview

This is a thought-oriented language model inspired by Deepseek-R1, trained via SFT and minimal RL on creative writing data, significantly improving instruction following, role-playing, and creative control abilities

Model Features

Thought Mode Activation

Activate thought mode by adding <think> tags before assistant responses to enhance reasoning capabilities

Creative Writing Optimization

Specifically optimized for creative writing and role-playing scenarios

High-quality Data Filtering

Utilizes rigorously screened and reconstructed semi-synthetic dialogue/role-playing datasets

Multi-source Data Integration

Incorporates human-annotated data, LLM-generated data, and Deepseek-R1's reasoning trajectory data

Model Capabilities

Instruction following

Role-playing

Creative writing

Natural conversation

Reasoning

Use Cases

Creative writing

Novel writing

Assisting authors in generating creative texts and storylines

Capable of producing coherent and creative narrative texts

Role-playing

Simulating dialogues and interactions of different characters

Maintains character consistency and generates dialogues that align with character traits

Dialogue systems

Intelligent assistant

Providing natural and fluent conversational experiences as an intelligent assistant

Capable of understanding complex instructions and providing logical responses

license: llama3.3 base_model:

meta-llama/Llama-3.3-70B-Instruct language:
en library_name: transformers

🦊 L3.3-70B-Vulpecula 🌸

Hi hi! 🌟

This is a collaboration work between GradientPutri and Sao10K.

This is a passion project of mine spanning the past few weeks, so we hope you like it.

While there may some minor issues, I think the final result is nice, and there are nice outputs which was the main goal.

Model card made by GradientPutri.

📜 Licensing Information

This model is based on Meta's Llama 3.3 and is subject to the Llama 3.3 Community License Agreement and the Acceptable Use Policy.

While we are unable to disallow commercial usage, do note that this is a project made using our own resources, time and effort. I'd rather not be discouraged from doing future project models instead. We kindly request that commercial users reach out before deployment to discuss usage and proper attribution. We appreciate users who help maintain transparency in the AI ecosystem by keeping us informed of how our work is being used. Same goes for any merges or derivatives, hopefully :)

🚀 Model Details

🌟 A thinking-based model inspired by Deepseek-R1, trained through both SFT and a little bit of RL on creative writing data.
🧠 Prefill, or begin assistant replies with <think>\n to activate thinking mode, or not. It works well without thinking too.
🚀 Improved Steerability, instruct-roleplay and creative control over base model.

📚 Dataset Composition

👾 Semi-synthetic Chat/Roleplaying datasets that has been re-made, cleaned and filtered for repetition, quality and output.
🎭 Human-based Natural Chat / Roleplaying datasets cleaned, filtered and checked for quality.
📝 Diverse Instruct dataset from a few different LLMs, cleaned and filtered for refusals and quality.
💭 Reasoning Traces taken from Deepseek-R1 for Instruct, Chat & Creative Tasks, filtered and cleaned for quality.
█▓▒ Toxic / Decensorship data was not needed for our purposes, the model is unrestricted enough as is.

Total token count: ~270M Tokens (210M Trainable), over 2 epochs.

🎨 Formatting and Samplers

Instruct Format: Llama-3-Instruct

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>

{input}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{output}<|eot_id|>
---
Note that newlines are represented within example above

✨ Sampler Recommendations

temperature: 0.75
min_p: 0.1
Repetition Penalty: 1.1
Presence Penalty: 1.1

⚙️ Training Details

# Iterations
num_epochs: 2

# Batching - Global Batch 4x GPUs × Batch 2 × 4 Grad_accum = 32
gradient_accumulation_steps: 4
micro_batch_size: 2

# Optimizer
optimizer: paged_ademamix_8bit
lr_scheduler: cosine
learning_rate: 0.00002
max_grad_norm: 1
weight_decay: 0.01

🦊 Thank you for visiting! May the foxes bring you good fortune! 🌸

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご