Laser Dolphin Mixtral 2x7b Dpo

🚀 Laser-Dolphin-Mixtral-2x7b-dpo

A medium-sized MoE implementation for text generation, based on a pre - trained model, with improved evaluation performance.

🚀 Quick Start

Using Ollama

ollama run macadeliccc/laser-dolphin-mixtral-2x7b-dpo

Using Python

from transformers import AutoModelForCausalLM, AutoTokenizer

def generate_response(prompt):
    """
    Generate a response from the model based on the input prompt.

    Args:
    prompt (str): Prompt for the model.

    Returns:
    str: The generated response from the model.
    """
    # Tokenize the input prompt
    inputs = tokenizer(prompt, return_tensors="pt")

    # Generate output tokens
    outputs = model.generate(**inputs, max_new_tokens=256, eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id)

    # Decode the generated tokens to a string
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)

    return response

# Load the model and tokenizer
model_id = "macadeliccc/laser-dolphin-mixtral-2x7b-dpo"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True)

prompt = "Write a quicksort algorithm in python"

# Generate and print responses for each language
print("Response:")
print(generate_response(prompt), "\n")

You can also check the colab for usage examples.

✨ Features

• Improved Performance: The new version shows ~1 point increase in evaluation performance on average.
• Multiple Quantizations: Available in ExLlamav2, GGUF, AWQ quantizations.

📦 Installation

Not provided in the original document.

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

def generate_response(prompt):
    """
    Generate a response from the model based on the input prompt.

    Args:
    prompt (str): Prompt for the model.

    Returns:
    str: The generated response from the model.
    """
    # Tokenize the input prompt
    inputs = tokenizer(prompt, return_tensors="pt")

    # Generate output tokens
    outputs = model.generate(**inputs, max_new_tokens=256, eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id)

    # Decode the generated tokens to a string
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)

    return response

# Load the model and tokenizer
model_id = "macadeliccc/laser-dolphin-mixtral-2x7b-dpo"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True)

prompt = "Write a quicksort algorithm in python"

# Generate and print responses for each language
print("Response:")
print(generate_response(prompt), "\n")

Advanced Usage

Switch the commented model definition to use in 4 - bit. Should work with 9GB and still exceed the single 7B model by 5 - 6 points roughly. The code is the same as the basic usage in this case, but you can adjust the parameters according to your needs.

📚 Documentation

Overview

This model is a medium - sized MoE implementation based on cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser. The new version shows ~1 point increase in evaluation performance on average.

Process

• The process is outlined in this notebook.
• The mergekit_config is in the files.
• The models used in the configuration are not lasered, but the final product is. This is an update from the last version.
• This process is experimental. Your mileage may vary.

Future Goals

• [ ] Function Calling
• [ ] v2 with new base model to improve performance

Quantizations

ExLlamav2

These are the recommended quantizations for users that are running the model on GPU. Thanks to user bartowski we now have exllamav2 quantizations in 3.5 through 8 bpw. They are available here:

• bartowski/laser-dolphin-mixtral-2x7b-dpo-exl2

His quantizations represent the first ~13B model with GQA support. Check out his repo for more information!

GGUF

Current GGUF Quantizations

AWQ

Current AWQ Quantizations

TheBloke

These Quants will result in unpredicted behavior. New quants are available as I have updated the model. Quatizations provided by TheBloke

HF Spaces

• GGUF chat available here
• 4 - bit bnb chat available here

Eval

EQ Bench

----Benchmark Complete----
2024-01-31 16:55:37
Time taken: 31.1 mins
Prompt Format: ChatML
Model: macadeliccc/laser-dolphin-mixtral-2x7b-dpo-GGUF
Score (v2): 72.76
Parseable: 171.0
---------------
Batch completed
Time taken: 31.2 mins
---------------

You can check the evaluation colab

Summary of previous evaluation

Detailed current evaluation

AGIEval

Average: 42.25%

GPT4All

Average: 73.45%

GSM8K

TruthfulQA

Average: 63.44%

Bigbench

🔧 Technical Details

Not provided in the original document.

📄 License

This project is licensed under the Apache - 2.0 license.