Llama 3 70b Arimas Story RP V1.6 4.5bpw H6 Exl2

🚀 EXL2 quants of ryzen88/Llama-3-70b-Arimas-story-RP-V1.6

This repository provides EXL2 quantized versions of the Llama-3-70b-Arimas-story-RP-V1.6 model, offering different bit - per - weight options for various usage scenarios.

🚀 Quick Start

Here are the links to different EXL2 quantized versions of the model:

• 3.00 bits per weight
• 3.50 bits per weight
• 4.00 bits per weight
• 4.50 bits per weight
• 6.00 bits per weight
• 8.00 bits per weight

These quantized models were created using the defaults from exllamav2 1.4.0 convert.py with the following settings:

• 3.0bpw to 6.0bpw head bits = 6
• 8.0bpw head bits = 8
• length = 8192
• dataset rows = 200
• measurement rows = 32
• measurement length = 8192

✨ Features

Model Overview

The base model is Llama-3-70b-Arimas-story-RP-V1.6, which is a merge of pre - trained language models created using mergekit.

Merge Details

• Experimentation: Greatly expanded the number of models used in this merge and experimented with various ideas. This version is more convincing than V1.5, and it is hoped that the long context window will remain strong after quantization.
• Data Type Switch: Due to the many merges, switched back from BFloat to Float.
• Failed Experiment: Tried breadcrumbs without the Ties, but the result was very poor.

Merge Method

This model was merged using the breadcrumbs_ties merge method, with I:\Llama-3-70B-Instruct-Gradient-262k as the base.

Models Merged

The following models were included in the merge:

• \Smaug-Llama-3-70B-Instruct
• \Meta-LLama-3-Cat-Smaug-LLama-70b
• \Meta-LLama-3-Cat-A-LLama-70b
• \Llama-3-70B-Synthia-v3.5
• \Llama-3-70B-Instruct-Gradient-524k
• \Llama-3-70B-Instruct-Gradient-262k
• \Tess-2.0-Llama-3-70B-v0.2
• \Llama-3-Lumimaid-70B-v0.1-alt

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: \Llama-3-70B-Instruct-Gradient-262k
    parameters:
      weight: 0.25
      density: 0.90
      gamma: 0.01
  - model: \Meta-LLama-3-Cat-Smaug-LLama-70b
    parameters:
      weight: 0.28
      density: 0.90
      gamma: 0.01
  - model: \Llama-3-Lumimaid-70B-v0.1-alt
    parameters:
      weight: 0.15
      density: 0.90
      gamma: 0.01
  - model: \Tess-2.0-Llama-3-70B-v0.2
    parameters:
      weight: 0.06
      density: 0.90
      gamma: 0.01
  - model: \Smaug-Llama-3-70B-Instruct
    parameters:
      weight: 0.04
      density: 0.90
      gamma: 0.01
  - model: \Llama-3-70B-Synthia-v3.5
    parameters:
      weight: 0.05
      density: 0.90
      gamma: 0.01
  - model: \Llama-3-70B-Instruct-Gradient-524k
    parameters:
      weight: 0.03
      density: 0.90
      gamma: 0.01
  - model: \Meta-LLama-3-Cat-A-LLama-70b
    parameters:
      weight: 0.14
      density: 0.90
      gamma: 0.01
merge_method: breadcrumbs_ties
base_model: I:\Llama-3-70B-Instruct-Gradient-262k
dtype: float16