Wanabi 24b Preview Gguf

🚀 wanabi-24B (preview)

wanabi-24B is a preview version of a large language model fine-tuned specifically for novel writing support.

This model is based on mistralai/Mistral-Small-24B-Base-2501 and trained using Japanese novel-related text data. It excels at tasks such as generating novel ideas, creating text based on settings, and generating subsequent content according to the context.

Notes on the Alpha Version:

• Proof of Concept: This version is intended for functionality verification.
• Limited Training: Training was conducted for only 1500 steps (plus 2000 steps with a batch size of 24) within the dataset.
• Provided Format: Currently, only the GGUF (Q4_K_M) format is provided.
• Characteristics: With only 1500 steps of fine-tuning, the extensive knowledge of the base model remains prominent (both positively and negatively). In a sense, it promises to be the most "knowledgeable" model among future versions. As novelists, let's look forward to future iterations.

Please stay tuned for future improvements.

🚀 Integration with Project Wannabe

We strongly recommend using this model in conjunction with the dedicated desktop application Project Wannabe. Project Wannabe provides a GUI to fully leverage the capabilities of wanabi-24B, seamlessly supporting tasks from idea generation to text writing and continuous (infinite) generation.

By using Project Wannabe, you can utilize the model's features without worrying about the prompt formats described below.

🔧 Technical Details

Base Model

• mistralai/Mistral-Small-24B-Base-2501
  • (During training, unsloth/Mistral-Small-24B-Base-2501-bnb-4bit was used)

Training Framework

• LLaMA-Factory

Training Method

• QLoRA (4-bit)
  • lora_rank: 128
  • lora_alpha: 256
  • lora_dropout: 0
  • lora_target: all (all linear layers)
• Precision: bf16
• Optimization:
  • PagedAdamW (8-bit)
  • Flash Attention 2
  • Unsloth Gradient Checkpointing (use_unsloth_gc: true)
  • Liger Kernel (enable_liger_kernel: true)
• Training Parameters:
  • learning_rate: 3.0e-5
  • lr_scheduler_type: cosine_with_restarts (num_cycles: 5)
  • warmup_ratio: 0.03
• Others:
  • cutoff_len: 32768
  • per_device_train_batch_size: 1
  • gradient_accumulation_steps: 24

📚 Documentation

Training Data and Tasks

Instruction Tuning (SFT) was performed using Japanese novel-related text data in the following three main task formats:

1. Text Generation (GEN):

   • Purpose: Generate novel text based on instructions and optionally provided metadata (title, keywords, genre, synopsis, settings, plot).
   • Example Format (with Metadata):

     <s>[INST] Please generate a novel text based on the following information.
     
     # Title:
     I Was Reincarnated as a Stray Dog in Another World
     
     # Keywords:
     Reincarnation in Another World
     Dog [/INST] {Generated Text} </s>
     

   • Example Format (without Metadata):

     <s>[INST] Please freely generate a novel. [/INST] {Generated Text} </s>
     

2. Continuation Generation (CONT):

   • Purpose: Generate the continuation of a given text, optionally referring to provided metadata.
   • Example Format (with Metadata):

     <s>[INST] Please generate the continuation of the following text based on the reference information.
     
     【Text】
     ```
     I thought I died after being stabbed by a stray attacker, but then I was reincarnated as a stray dog in another world.
     ```
     【Reference Information】
     ```
     # Title:
     I Was Reincarnated as a Stray Dog in Another World
     
     # Keywords:
     Reincarnation in Another World
     Dog
     Exile
     ``` [/INST] {Generated Continuation Text} </s>
     

   • Example Format (without Metadata):

     <s>[INST] Please generate the continuation of the following text.
     
     【Text】
     ```
     I thought I died after being stabbed by a stray attacker, but then I was reincarnated as a stray dog in another world.
     ``` [/INST] {Generated Continuation Text} </s>
     

3. Idea Generation (IDEA):

   • Purpose: Generate a complete novel idea (title, keywords, genre, synopsis, settings, plot) from optionally provided partial (or no) metadata.
   • Example Format (with Partial Metadata):

     <s>[INST] Please generate a complete novel idea (title, keywords, genre, synopsis, settings, plot) based on the following information.
     
     # Keywords:
     Reincarnation in Another World
     Dog [/INST] # Title:
     I Was Reincarnated as a Stray Dog in Another World
     
     # Keywords:
     Reincarnation in Another World
     Dog
     Exile
     Romance
     NTR
     
     # Genre:
     Another World Fantasy
     Low Fantasy
     
     # Synopsis:
     I thought I died after being stabbed by a stray attacker, but then I was reincarnated as a stray dog in another world. At first, I was in despair, but then I was picked up by a kind girl...
     
     # Settings:
     The story is set in a medieval-style other world of swords and magic. The protagonist has modern knowledge but is trapped in a dog's body.
     
     # Plot:
     Meeting the girl -> Commotion in the town -> Awakening of the protagonist's special ability -> Crisis of exile -> ... </s>
     

   • Example Format (without Metadata):

     <s>[INST] Please freely generate a novel idea (title, keywords, genre, synopsis, settings, plot). [/INST] {Generated Set of Ideas} </s>
     

Prompt Template: The mistral_small template format was used during training. We also recommend using the same format (<s>[INST] {instruction} {input} [/INST] {output} </s>) during inference.

⚠️ Limitations and Precautions

⚠️ Important Note

• Alpha Version: This model is a preview version under development, and its performance and stability are not guaranteed.
• Bias: Due to the characteristics of the training data, the generated content may be biased towards specific genres, expressions, and developments.
• Inappropriate Content: Since the training data contains diverse texts, the model may generate text that is unsuitable for minors or may cause discomfort.
• Quality Limitations: There are limitations to the diversity, consistency, and context-following ability of the generated text. In particular, generating long texts may result in inconsistencies.
• Usage Precautions: This model is provided for research and experimental purposes. Users are responsible for complying with applicable laws and regulations. Use for illegal purposes or to infringe on the rights of others is strictly prohibited.
• Self-Responsibility: The developers assume no responsibility for any consequences arising from the use of this model. Please use it at your own risk.

Future Plans

• The additional training version from the preview version has been abolished. The preview_plus is the final model of the same dataset.
• Currently training with a dataset that supports ratings, dialogue volume, and author's notes.
• wanabi-24B vX: SFT with an extended dataset is in progress (to be released gradually).

(The roadmap is subject to change)

📄 License

The license of this model is Apache-2.0 (subject to the base model, please change as needed).