Llama 3 70b Arimas Story RP V1.6 3.5bpw H6 Exl2

🚀 EXL2 Quants of ryzen88/Llama-3-70b-Arimas-story-RP-V1.6

This project provides EXL2 quantization versions of the ryzen88/Llama-3-70b-Arimas-story-RP-V1.6 model. These quantized models offer different bits per weight options, which can help in optimizing the model's memory usage and inference speed.

Quantized Model Links

• 3.00 bits per weight
• 3.50 bits per weight
• 4.00 bits per weight
• 4.50 bits per weight
• 6.00 bits per weight
• 8.00 bits per weight

Quantization Details

• Created using the defaults from exllamav2 1.4.0 convert.py
• 3.0bpw to 6.0bpw head bits = 6
• 8.0bpw head bits = 8
• length = 8192
• dataset rows = 200
• measurement rows = 32
• measurement length = 8192

✨ Features

• Multiple Quantization Options: Offers a range of bits per weight options to suit different memory and performance requirements.
• Merge of Pre-trained Models: Created by merging multiple pre-trained language models using mergekit, which potentially combines the strengths of different models.

📚 Documentation

Model Overview

The Llama-3-70b-Arimas-story-RP-V1.6 model is a merge of several pre-trained language models. This merge aims to create a model with enhanced performance and capabilities.

Merge Details

• Experimentation: Greatly expanded the number of models used in this merge and experimented with various ideas. This version feels more convincing than V1.5, and it is hoped that the long context window will remain strong after quantization.
• Data Type Change: Switched back from BFloat to Float due to the many merges.
• Failed Experiment: Tried breadcrumbs without the Ties, which did not work well.

Merge Method

This model was merged using the breadcrumbs_ties merge method, with I:\Llama-3-70B-Instruct-Gradient-262k as the base model.

Models Merged

The following models were included in the merge:

• \Smaug-Llama-3-70B-Instruct
• \Meta-LLama-3-Cat-Smaug-LLama-70b
• \Meta-LLama-3-Cat-A-LLama-70b
• \Llama-3-70B-Synthia-v3.5
• \Llama-3-70B-Instruct-Gradient-524k
• \Llama-3-70B-Instruct-Gradient-262k
• \Tess-2.0-Llama-3-70B-v0.2
• \Llama-3-Lumimaid-70B-v0.1-alt

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: \Llama-3-70B-Instruct-Gradient-262k
    parameters:
      weight: 0.25
      density: 0.90
      gamma: 0.01
  - model: \Meta-LLama-3-Cat-Smaug-LLama-70b
    parameters:
      weight: 0.28
      density: 0.90
      gamma: 0.01
  - model: \Llama-3-Lumimaid-70B-v0.1-alt
    parameters:
      weight: 0.15
      density: 0.90
      gamma: 0.01
  - model: \Tess-2.0-Llama-3-70B-v0.2
    parameters:
      weight: 0.06
      density: 0.90
      gamma: 0.01
  - model: \Smaug-Llama-3-70B-Instruct
    parameters:
      weight: 0.04
      density: 0.90
      gamma: 0.01
  - model: \Llama-3-70B-Synthia-v3.5
    parameters:
      weight: 0.05
      density: 0.90
      gamma: 0.01
  - model: \Llama-3-70B-Instruct-Gradient-524k
    parameters:
      weight: 0.03
      density: 0.90
      gamma: 0.01
  - model: \Meta-LLama-3-Cat-A-LLama-70b
    parameters:
      weight: 0.14
      density: 0.90
      gamma: 0.01
merge_method: breadcrumbs_ties
base_model: I:\Llama-3-70B-Instruct-Gradient-262k
dtype: float16