Capytessborosyi 34B 200K DARE Ties

🚀 CaPlatTessDolXaBoros-Yi-34B-200K-DARE-Ties-HighDensity

This project merges multiple language models using a new experimental implementation of "dare ties" to achieve better text - generation performance.

🚀 Quick Start

This model is obsolete. Please refer to: https://huggingface.co/brucethemoose/CaPlatTessDolXaBoros-Yi-34B-200K-DARE-Ties-HighDensity

✨ Features

NousResearch/Nous-Capybara-34B, migtissera/Tess-M-v1.3, and bhenrym14/airoboros-3_1-yi-34b-200k are merged using a new experimental implementation of "dare ties" via mergekit. For more details, refer to the paper:

Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch https://github.com/yule-BUAA/MergeLM https://github.com/cg123/mergekit/tree/dare

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Prompt template: Orca - Vicuna

SYSTEM: {system_message}
USER: {prompt}
ASSISTANT:

⚠️ Important Note

As it is a Yi model, if the output doesn't seem right, try disabling the BOS token and/or running at a lower temperature with MinP (and no other samplers). Yi models tend to run "hot" by default.

Sometimes the model "spells out" the stop token as </s> like Capybara, so you may need to add </s> as an additional stopping condition. It might also respond to the llama - 2 chat format.

GPU Usage

24GB GPUs can run Yi - 34B - 200K models at 45K - 75K context with exllamav2. For more details, check this post. It is recommended to use exl2 quantizations on data similar to the desired task, such as these targeted at story writing: 4.0bpw / 3.1bpw

🔧 Technical Details

The models are merged with the following config, and the tokenizer from chargoddard's Yi - Llama:

models:
  - model: /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama
    # no parameters necessary for base model
  - model: /home/alpha/Storage/Models/Raw/migtissera_Tess-M-v1.3
    parameters:
      weight: 0.41
      density: 0.50
  - model: /home/alpha//Storage/Models/Raw/bhenrym14_airoboros-3_1-yi-34b-200k
    parameters:
      weight: 0.18
      density: 0.46
  - model: /home/alpha/Storage/Models/Raw/Nous-Capybara-34B
    parameters:
      weight: 0.41
      density: 0.50
merge_method: dare_ties
base_model: /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama
parameters:
int8_mask: true
dtype: bfloat16

The "dare ties" method shows better perplexity than a regular ties merge with the same merge configuration. Testing also indicates that model weights adding up to one seem optimal, and high - context results seem better than the previous dare merge with Tess 1.2.

📄 License

• License: other
• License Name: yi - license
• License Link: https://huggingface.co/01-ai/Yi-34B/blob/main/LICENSE

Credits

• https://github.com/cg123/mergekit/tree/dare
• https://huggingface.co/NousResearch/Nous-Capybara-34B/
• https://huggingface.co/bhenrym14/airoboros-3_1-yi-34b-200k
• https://huggingface.co/migtissera/Tess-M-v1.3
• https://huggingface.co/chargoddard/Yi-34B-200K-Llama
• https://huggingface.co/01-ai/Yi-34B-200K