Razorback 12B V0.2

🚀 Razorback 12B v0.2

UnslopNemo with Vision!

This project presents a more robust attempt at merging TheDrummer's UnslopNemo v3 into Pixtral 12B. It combines the strengths of both models, offering less sloppy, more engaging content along with decent intelligence and visual understanding.

🚀 Quick Start

The model has been quite stable in initial testing, but more testing is needed to identify the best sampling parameters. So far, using a temperature of ~0.7 and a minimum p of 0.03 has proven to be reliable. Use the included chat template (Mistral), as there is no chatml support yet.

✨ Features

• Multilingual Support: Supports multiple languages including English, French, German, Spanish, Italian, Portuguese, Russian, Chinese, and Japanese.
• Image-Text-to-Text: The pipeline tag indicates its ability to handle image and text inputs for text generation.
• Combined Strengths: Merges the characteristics of Pixtral 12B and UnslopNemo-12B-v3, offering less sloppy and more engaging content.

📦 Installation

No specific installation steps were provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

# No code example was provided in the original document.
# More testing is needed to identify the best sampling params.
# So far, using ~0.7 temp + 0.03 min p has been reliable.
# Use the included chat template (Mistral).

📚 Documentation

Merging Approach

First, the Pixtral 12B Base and Mistral Nemo Base were loaded to compare their parameter differences. By looking at the L2 norm and relative difference values, the parts of Pixtral 12B that significantly deviate from Mistral Nemo were isolated. Although the language model architecture is the same between the two, a lot of vision understanding has been trained into Pixtral's language model, which can break easily.

Then, merging weights for each parameter were calculated using an exponential falloff. The smaller the difference, the higher the weight. This recipe was applied to Pixtral Instruct (Pixtral-12B-2409) and TheDrummer's UnslopNemo-12B-v3. The goal was to infuse as much of the Drummer model's goodness without breaking vision input, and it seems to have worked!

🔧 Technical Details

The merging process involved comparing parameter differences between Pixtral 12B Base and Mistral Nemo Base, isolating significant deviations, and calculating merging weights using an exponential falloff. This approach aimed to combine the two models while preserving vision input functionality.

📄 License

The license for this project is classified as "other".

📦 Model Information

👏 Credits

• Mistral for mistralai/Pixtral-12B-2409
• Unsloth for unsloth/Pixtral-12B-2409 transformers conversion
• TheDrummer for TheDrummer/UnslopNemo-12B-v3