Wanabi Mini 12b GGUF

🚀 wanabi_mini_12b_GGUF

wanabi_mini_12b_GGUF is a Japanese large language model fine-tuned specifically for novel writing support. We've prepared a model that is easier for more users to handle while having the same functionality as wanabi-24B.

This model is based on mistralai/Mistral-Nemo-Base-2407. Although the dataset scale is smaller than the 24B version, it has been trained using higher-quality Japanese novel-related text data. It aims to support various processes in novel writing, from idea generation to text generation, generating continuations that follow the context, and even idea interpolation.

• Provided format: Currently, only the GGUF format is provided. Multiple quantized versions suitable for consumer GPUs with 8GB or more of VRAM are available. All quantized models are imatrix-quantized using calibration data of their characteristics.
• Features: Compared to the 24B version, it has been trained on a higher-quality dataset, and improvements in responsiveness and accuracy for specific tasks are expected.

🚀 Quick Start

Integration with Project Wannabe (Highly Recommended)

It is highly recommended to use this model in conjunction with the dedicated desktop application Project Wannabe. Project Wannabe provides a GUI to fully leverage the capabilities of wanabi_mini_12b_GGUF and is designed to allow intuitive use of the features described below.

✨ Features

New Features (Compared to wanabi-24B v1)

In addition to the main features of the 24B version, wanabi_mini_12b_GGUF has the following new features:

1. Idea Interpolation Function (New):
   • Purpose: In the "Detailed Information" tab of Project Wannabe, when all the items necessary for a novel idea (title, keywords, genre, synopsis, setting, plot) are entered, it generates more detailed and in-depth ideas and development hints based on that information.
   • Application: It is activated when specific conditions are met in the idea generation (IDEA) task.

Main Features

It provides the same basic novel writing support functions as wanabi-24B.

1. Author's Note Function:

   • Purpose: By describing the upcoming content, such as the next development, actions, and mood descriptions, within approximately 1000 characters, it guides the generation of the subsequent text in more detail.
   • Application: It is incorporated into the prompt for the continuation generation (CONT) task.

2. Rating Function:

   • Purpose: Specifies the rating (general or r18) of the generated content.
   • Application: Rating: {specified value} is added to the end of the instruction for all tasks (GEN, CONT, IDEA).

3. Dialogue Quantity Specification Function:

   • Purpose: Selects the proportion of dialogue in the generated text from options such as "unspecified", "low", "slightly low", "normal", "slightly high", and "high". (This feature is not fully implemented in the current version but is planned for future versions.)
   • Application: When an option other than "unspecified" is selected, # Dialogue Quantity: {specified value} is included in the input part (within the reference information block) of the prompt for the text generation (GEN) and continuation generation (CONT) tasks.

4. Text Generation (GEN):

   • Based on the given instructions, optional metadata (title, keywords, genre, synopsis, setting, plot), dialogue quantity, and rating, it generates the text of the novel.

5. Continuation Generation (CONT):

   • Generates the continuation of the given text, taking into account the optional metadata, dialogue quantity, rating, and author's note.
   • The prompt structure is in an improved format similar to wanabi-24B v0.1.

6. Idea Generation (IDEA):

   • Based on part (or none) of the optional metadata and rating, it generates novel ideas (title, keywords, genre, synopsis, setting, plot).
   • With the idea interpolation function, more detailed ideas are generated when the input information is rich.

🔧 Technical Details

Base Model

• mistralai/Mistral-Nemo-Base-2407

Training Framework

• LLaMA-Factory

Training Method

• Method: Supervised Fine-tuning (SFT)
• Quantization and Adapter: LoRA
  • lora_rank: 128
  • lora_alpha: 256
  • lora_dropout: 0.05
  • lora_target: all (all linear layers)
• Precision: bf16
• Sequence Length: 32768
• Batch Size: per_device_train_batch_size = 1, gradient_accumulation_steps = 24 (effective batch size 24)
• Optimization:
  • Optimizer: PagedAdamW (8-bit) (optim: paged_adamw_8bit)
  • Flash Attention 2: Enabled (flash_attn: fa2)
  • Unsloth Gradient Checkpointing: Enabled (use_unsloth_gc: true)
  • Liger Kernel: Enabled (enable_liger_kernel: true)
  • Weight Decay: 0.01 (weight_decay: 0.01)
• Learning Rate:
  • learning_rate: 4.0e-5
  • lr_scheduler_type: cosine_with_restarts
  • lr_scheduler_kwargs: {"num_cycles": 1}
  • warmup_ratio: 0.03
• Others:
  • num_train_epochs: 1

📚 Documentation

Prompt Format (mistral_small Template)

This model has been trained using the mistral_small chat template format of LLaMA-Factory. The same format is recommended for inference. When using Project Wannabe, you don't need to worry about it. Since the basic format is the same as wanabi-24B, the details are omitted.

• New Feature: Idea Interpolation:

  When generating ideas in the "Detailed Information" tab of Project Wannabe with all the title, keywords, genre, synopsis, setting, and plot entered, the model attempts to generate more detailed and specific ideas (e.g., in-depth character analysis, sub-plot suggestions, supplementary explanations of the world view) by leveraging this rich information. This feature does not require any special prompt changes and automatically adapts based on the quantity and quality of the input information.

⚠️ Important Note

• Model in Development: This model is currently under development, and its performance and stability may improve in future versions.
• Bias: Due to the characteristics of the training data, the generated content may be biased towards specific genres, expressions, and developments.
• Inappropriate Content: Since the training data contains diverse texts, the model may generate texts that could cause discomfort. Although the rating function attempts to control this, it is not perfect.
• Quality Limitations: There are limitations to the diversity, consistency, and context-following ability of the generated text.
• Usage Precautions: This model is provided for research and experimental purposes. Its use for illegal purposes or to infringe on the rights of others is strictly prohibited.
• User Responsibility: The developers assume no responsibility for any results arising from the use of this model.

📄 License

This model complies with the Apache-2.0 license.