Llama 3 70b Arimas Story RP V1.6 3.0bpw H6 Exl2

🚀 EXL2 quants of ryzen88/Llama-3-70b-Arimas-story-RP-V1.6

This project provides EXL2 quantization versions of the Llama-3-70b-Arimas-story-RP-V1.6 model, offering multiple bit - per - weight options.

🚀 Quick Start

Here are the links to different bit - per - weight EXL2 quantizations of the model:

• 3.00 bits per weight
• 3.50 bits per weight
• 4.00 bits per weight
• 4.50 bits per weight
• 6.00 bits per weight
• 8.00 bits per weight

These quantizations were created using the defaults from exllamav2 1.4.0 convert.py.

• For 3.0bpw to 6.0bpw, head bits = 6
• For 8.0bpw, head bits = 8
• length = 8192
• dataset rows = 200
• measurement rows = 32
• measurement length = 8192

✨ Features

Model Introduction

The model is Llama-3-70b-Arimas-story-RP-V1.6, which is a merge of pre - trained language models created using mergekit.

Merge Details

• Experiment and Improvement: I greatly expanded the number of models used in this merge and experimented extensively with different ideas. This version seems more convincing than V1.5. Hopefully, the long context window will remain strong even after quantization.
• Data Type Change: Due to the numerous merges, I switched back from BFloat to Float.
• Failed Experiment: An attempt to use breadcrumbs without ties turned out very poorly.

Merge Method

This model was merged using the breadcrumbs_ties merge method, with I:\Llama-3-70B-Instruct-Gradient-262k as the base.

Models Merged

The following models were included in the merge:

• \Smaug-Llama-3-70B-Instruct
• \Meta-LLama-3-Cat-Smaug-LLama-70b
• \Meta-LLama-3-Cat-A-LLama-70b
• \Llama-3-70B-Synthia-v3.5
• \Llama-3-70B-Instruct-Gradient-524k
• \Llama-3-70B-Instruct-Gradient-262k
• \Tess-2.0-Llama-3-70B-v0.2
• \Llama-3-Lumimaid-70B-v0.1-alt

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: \Llama-3-70B-Instruct-Gradient-262k
    parameters:
      weight: 0.25
      density: 0.90
      gamma: 0.01
  - model: \Meta-LLama-3-Cat-Smaug-LLama-70b
    parameters:
      weight: 0.28
      density: 0.90
      gamma: 0.01
  - model: \Llama-3-Lumimaid-70B-v0.1-alt
    parameters:
      weight: 0.15
      density: 0.90
      gamma: 0.01
  - model: \Tess-2.0-Llama-3-70B-v0.2
    parameters:
      weight: 0.06
      density: 0.90
      gamma: 0.01
  - model: \Smaug-Llama-3-70B-Instruct
    parameters:
      weight: 0.04
      density: 0.90
      gamma: 0.01
  - model: \Llama-3-70B-Synthia-v3.5
    parameters:
      weight: 0.05
      density: 0.90
      gamma: 0.01
  - model: \Llama-3-70B-Instruct-Gradient-524k
    parameters:
      weight: 0.03
      density: 0.90
      gamma: 0.01
  - model: \Meta-LLama-3-Cat-A-LLama-70b
    parameters:
      weight: 0.14
      density: 0.90
      gamma: 0.01
merge_method: breadcrumbs_ties
base_model: I:\Llama-3-70B-Instruct-Gradient-262k
dtype: float16

📚 Documentation

Model Information Table