MN - Slush Open - Source AI Model - Free Deployment, Strongly Enhance Creativity and Role

MN Slush

Developed by crestf411

Slush is a two-stage model trained with high LoRA dropout rate, focusing on enhancing creativity and role-playing capabilities

Large Language Model

Transformers

#High LoRA Dropout Rate Training #Role-playing Optimization #Long Context Support

Downloads 59

Release Time : 11/20/2024

Model Overview

Slush is a specially trained large language model that adopts a two-stage training approach. The first stage continues the pre-training of the base model to improve creativity and writing abilities, while the second stage involves fine-tuning to enhance role-playing capabilities. The model is suitable for non-all-age users and is particularly ideal for role-playing scenarios.

Model Features

Two-stage Training

Adopts pre-training continuation and fine-tuning stages to enhance creativity and role-playing capabilities respectively

High LoRA Dropout Rate

Uses a LoRA dropout rate of 0.5 during training, optimizing model performance based on the latest research findings

Long Context Support

Supports long context processing of up to 16384 tokens

LoRA+ Technology

Employs LoRA+ training method with a learning rate ratio of 15 to optimize training effectiveness

Model Capabilities

Creative writing

Role-playing dialogue

Long text generation

Instruction following

Use Cases

Entertainment

Role-playing Games

Engages in immersive interactions as in-game characters with users

Delivers high-quality role-playing experiences

Creative Writing Assistance

Helps users generate creative stories or novel content

Produces creative text outputs

Dialogue Systems

Personalized Chatbots

Builds chatbots with specific personality traits

Provides personalized and consistent dialogue experiences

🚀 Slush

Slush is a two - stage model trained with high LoRA dropout. It aims to boost the model's creativity and writing capabilities in the first stage and further enhance its role - playing capabilities in the second stage.

🚀 Quick Start

This model is still in the early stage. Feedback is always welcome, but perfection - demanding users may not find it suitable. The second stage follows the Silly Tavern preset (Mistral V2 & V3, V3 - Tekken also works fine), so the performance may vary if you use other tools or presets.

✨ Features

Two - stage Training: Stage 1 is a pretraining continuation on the base model, and stage 2 is a fine - tuning step on top of the result of stage 1.
High LoRA Dropout: Helps in enhancing the model's generalization and performance.

📚 Documentation

Parameter Suggestions

All testing was done with temp 1, min - p 0.1, and DRY 0.8.

Training Details

Stage 1 (Continued Pretraining)

Target: mistralai/Mistral - Nemo - Base - 2407 (resulting LoRA merged into mistralai/Mistral - Nemo - Instruct - 2407)
LoRA Dropout: 0.5 (motivation)
LoRA Rank: 64, alpha 128 (motivation)
LR Cosine: 4e - 6
LoRA+ with LR Ratio: 15
Context Size: 16384
Gradient Accumulation Steps: 4
Epochs: 1

Stage 2 (Fine Tune)

Target: Stage 1 model
LoRA Dropout: 0.5
LoRA Rank: 32, alpha 64
LR Cosine: 5e - 6 (min 5e - 7)
LoRA+ with LR Ratio: 15
Context Size: 16384
Gradient Accumulation Steps: 4
Epochs: 2

Merge Details

Merge Method

This model was merged using the TIES merge method with mistralai/Mistral - Nemo - Base - 2407 as a base.

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: stage1-on-instruct
    parameters:
      weight: 1
      density: 1
  - model: stage2-on-stage1
    parameters:
      weight: 0.7
      density: 1
  - model: mistralai/Mistral-Nemo-Instruct-2407
    parameters:
      weight: 1
      density: 1
merge_method: ties
base_model: mistralai/Mistral-Nemo-Base-2407
parameters:
  weight: 1
  density: 1
  normalize: true
  int8_mask: true
tokenizer_source: mistralai/Mistral-Nemo-Instruct-2407
dtype: bfloat16

📄 License

No license information is provided in the original document.

📦 Information

Property	Details
Library Name	transformers
Tags	not - for - all - audiences, mergekit
Datasets	crestf411/LimaRP - DS, Gryphe/Sonnet3.5 - Charcard - Roleplay, anthracite - org/c2_logs_32k_mistral - v3_v1.2_no_system, anthracite - org/kalo - opus - instruct - 22k - no - refusal - no - system, anthracite - org/kalo - opus - instruct - 3k - filtered - no - system, anthracite - org/nopm_claude_writing_fixed
Base Model	mistralai/Mistral - Nemo - Instruct - 2407

(GGUFs)

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご