Beyonder 4x7B V2

🚀 Beyonder-4x7B-v2

Beyonder-4x7B-v2 is a Mixture of Experts (MoE) model created with mergekit, leveraging multiple base models to offer excellent text - generation capabilities.

This model is a Mixture of Experts (MoE) made with mergekit (mixtral branch). It uses the following base models:

• openchat/openchat-3.5-1210
• beowolx/CodeNinja-1.0-OpenChat-7B
• maywell/PiVoT-0.1-Starling-LM-RP
• WizardLM/WizardMath-7B-V1.1

The recommended context length is 8k.

✨ Features

• Mixture of Experts Architecture: Combines the strengths of multiple base models for enhanced performance.
• Quantized Models Available: Thanks to TheBloke, various quantized versions are accessible for different usage scenarios.
• Competitive Performance: Performs competitively with larger models on the Open LLM Leaderboard and shows significant improvement over individual experts.

📦 Installation

No specific installation steps are provided in the original README. If you want to use the model, you can refer to the usage example below.

💻 Usage Examples

Basic Usage

Here's a notebook to run this model in 4 - bit precision using a free T4 GPU on Google Colab.

!pip install -qU transformers bitsandbytes accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "mlabonne/Beyonder-4x7B-v2"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True},
)

messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])

Output:

A Mixture of Experts (ME) is a machine learning technique that combines multiple expert models to make predictions or decisions. Each expert model is specialized in a different aspect of the problem, and their outputs are combined to produce a more accurate and robust solution. This approach allows the model to leverage the strengths of individual experts and compensate for their weaknesses, improving overall performance.

🏆 Evaluation

Beyonder-4x7B-v2 is competitive with Mixtral-8x7B-Instruct-v0.1 on the Open LLM Leaderboard, while only having 4 experts instead of 8.

It also displays a significant improvement over the individual experts.

It also performs very well compared to other models on Nous benchmark suite. It's almost as good as the best Yi - 34B fine - tune, which is a much bigger model: 24.2B parameters + only two experts are selected during inference (so ~12B) vs. 34B param.

AGIEval

GPT4All

TruthfulQA

Bigbench

🧩 Configuration

base_model: mlabonne/Marcoro14-7B-slerp
experts:
  - source_model: openchat/openchat-3.5-1210
    positive_prompts:
    - "chat"
    - "assistant"
    - "tell me"
    - "explain"
  - source_model: beowolx/CodeNinja-1.0-OpenChat-7B
    positive_prompts:
    - "code"
    - "python"
    - "javascript"
    - "programming"
    - "algorithm"
  - source_model: maywell/PiVoT-0.1-Starling-LM-RP
    positive_prompts:
    - "storywriting"
    - "write"
    - "scene"
    - "story"
    - "character"
  - source_model: WizardLM/WizardMath-7B-V1.1
    positive_prompts:
    - "reason"
    - "math"
    - "mathematics"
    - "solve"
    - "count"

⚡ Quantized models

Thanks to TheBloke for the quantized models:

• GGUF: https://huggingface.co/TheBloke/Beyonder-4x7B-v2-GGUF
• AWQ: https://huggingface.co/TheBloke/Beyonder-4x7B-v2-AWQ
• GPTQ: https://huggingface.co/TheBloke/Beyonder-4x7B-v2-GPTQ
• EXL2: https://huggingface.co/bartowski/Beyonder-4x7B-v2-exl2

📄 License

This model is under the microsoft - research - license.