Chicka-Mixtral-3x7b Open-Source Large Model - Effortlessly Handle Conversations, Coding, and Math Tasks

Chicka Mixtral 3x7b

Developed by Chickaboo

A Mixture of Experts large language model based on 3 Mistral architecture models, excelling in dialogue, code, and mathematical tasks

Large Language Model

Transformers

Open Source License:MIT #Mixture of Experts #Multi-domain Dialogue #Enhanced Mathematical Reasoning

Downloads 77

Release Time : 4/22/2024

Model Overview

This model is a Mixture of Experts large language model based on 3 Mistral architecture models, containing expert modules for basic dialogue, code, and mathematics, capable of automatically switching to the optimal expert for different tasks

Model Features

Mixture of Experts Architecture

Integrates expert models from three specialized domains: dialogue, code, and mathematics, automatically selecting the optimal expert based on input content

Intelligent Trigger Mechanism

Automatically identifies task types through keywords and activates the corresponding expert module

High Performance

Outperforms similar 7B/8B scale models in multiple benchmark tests

Model Capabilities

Natural Language Dialogue

Code Generation and Explanation

Mathematical Problem Solving

Multi-turn Dialogue

Text Understanding and Generation

Use Cases

Development Assistance

Code Generation

Generates code in multiple programming languages based on natural language descriptions

Supports various languages including Python, JavaScript, C++

Code Debugging

Helps developers understand and fix code errors

Can explain runtime errors and provide solutions

Education

Math Tutoring

Solves mathematical problems and displays step-by-step solutions

Achieved a score of 70.66 on the GSM8K math test

Concept Explanation

Explains complex concepts in simple language

Suitable for learners with different knowledge levels

Intelligent Assistant

Daily Q&A

Answers various daily questions and provides suggestions

Achieved a score of 50.51 in real-world Q&A tests

Recipe Recommendation

Provides cooking suggestions and recipes based on user needs

Can generate detailed cooking steps

🚀 Chicka-Mistral-3x7b

This is a Mixture of Experts merged LLM, leveraging the strengths of multiple Mistral-based models for enhanced text generation capabilities.

🚀 Quick Start

This model is a powerful tool for text generation. You can quickly get started by following the usage examples below.

✨ Features

Mixture of Experts: Combines three Mistral-based models, each serving as an expert in different domains (conversational, code, and math).
High Performance: Demonstrates excellent performance on various benchmarks compared to other models.

📚 Documentation

Model Description

This model is a Mixture of Experts merged LLM consisting of 3 mistral based models:

Base model/conversational expert: openchat/openchat-3.5-0106
Code expert: beowolx/CodeNinja-1.0-OpenChat-7B
Math expert: meta-math/MetaMath-Mistral-7B

This is the Mergekit config used in the merging process:

base_model: openchat/openchat-3.5-0106
experts:
  - source_model: openchat/openchat-3.5-0106
    positive_prompts:
    - "chat"
    - "assistant"
    - "tell me"
    - "explain"
    - "I want"
  - source_model: beowolx/CodeNinja-1.0-OpenChat-7B
    positive_prompts:
    - "code"
    - "python"
    - "javascript"
    - "programming"
    - "algorithm"
    - "C#"
    - "C++"
    - "debug"
    - "runtime"
    - "html"
    - "command"
    - "nodejs"
  - source_model: meta-math/MetaMath-Mistral-7B
    positive_prompts:
    - "reason"
    - "math"
    - "mathematics"
    - "solve"
    - "count"
    - "calculate"
    - "arithmetic"
    - "algebra"

Open LLM Leaderboards

Benchmark	Chicka-Mixtral-3X7B	Mistral-7B-Instruct-v0.2	Meta-Llama-3-8B
Average	69.19	60.97	62.55
ARC	64.08	59.98	59.47
Hellaswag	83.96	83.31	82.09
MMLU	64.87	64.16	66.67
TruthfulQA	50.51	42.15	43.95
Winogrande	81.06	78.37	77.35
GSM8K	70.66	37.83	45.79

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained("Chickaboo/Chicka-Mistral-3x7b")
tokenizer = AutoTokenizer.from_pretrained("Chickaboo/Chicka-Mixtral-3x7b")

messages = [
    {"role": "user", "content": "What is your favourite condiment?"},
    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
    {"role": "user", "content": "Do you have mayonnaise recipes?"}
]

encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")

model_inputs = encodeds.to(device)
model.to(device)

generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])

📄 License

This project is licensed under the MIT License.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご