Orthogonal-2x7B-v2-base Open-source Text Generation Model - Free Deployment to Aid Diverse Text Creation

Orthogonal 2x7B V2 Base

Developed by LoSboccacc

orthogonal-2x7B-v2-base is a Mixture of Experts model based on Mistral-7B-Instruct-v0.2 and SanjiWatsuki/Kunoichi-DPO-v2-7B, specializing in text generation tasks.

Large Language Model

Transformers

#Multi-task reasoning #Few-shot learning #Role-play optimization

Downloads 80

Release Time : 1/18/2024

Model Overview

This model combines the capabilities of two expert models, each excelling in role-play and chat tasks respectively, dynamically selecting the most suitable expert model for text generation through a gating mechanism.

Model Features

Mixture of Experts architecture

Combines the capabilities of two 7B-parameter models, dynamically selecting the most suitable expert model through a gating mechanism.

Multi-task optimization

Utilizes specialized expert models for different tasks (e.g., role-play and chat) to improve generation quality.

Efficient inference

Compared to a single large model, the Mixture of Experts architecture can maintain performance while improving inference efficiency.

Model Capabilities

Text generation

Dialogue systems

Role-play

Question answering systems

Reasoning tasks

Use Cases

Dialogue systems

Intelligent chat assistant

Used to build natural and fluent dialogue systems

Achieved 85.69% accuracy on the HellaSwag dataset

Education

Scientific Q&A system

Answers science-related questions

Achieved 66.89% accuracy on the AI2 Reasoning Challenge

Mathematical reasoning

Math problem solving

Solves basic math problems

Achieved 51.4% accuracy on the GSM8k dataset

🚀 orthogonal-2x7B-v2-base

This project presents the orthogonal-2x7B-v2-base model, which is evaluated on multiple text - generation tasks. The evaluation results show its performance on various datasets, providing insights into its capabilities in different scenarios.

📚 Documentation

Model Information

Property	Details
Model Name	orthogonal-2x7B-v2-base
Base Model	mistralai/Mistral-7B-Instruct-v0.2
Gate Mode	hidden (one of "hidden", "cheap_embed", or "random")
Data Type	bfloat16 (output dtype: float32, float16, or bfloat16)

Experts Information

The model uses the following expert models with corresponding positive prompts:

Expert 1:
- Source Model: SanjiWatsuki/Kunoichi-DPO-v2-7B
- Positive Prompts: ["roleplay"]
Expert 2:
- Source Model: mistralai/Mistral-7B-Instruct-v0.2
- Positive Prompts: ["chat"]

Evaluation Results

The model is evaluated on multiple text - generation tasks. Here are the detailed results:

AI2 Reasoning Challenge (25 - Shot)

Task: Text Generation
Dataset: AI2 Reasoning Challenge (25 - Shot) (ai2_arc with config ARC - Challenge, test split, 25 few - shot)
Metric: Normalized Accuracy (acc_norm) = 66.89
Source: Open LLM Leaderboard

HellaSwag (10 - Shot)

Task: Text Generation
Dataset: HellaSwag (10 - Shot) (hellaswag, validation split, 10 few - shot)
Metric: Normalized Accuracy (acc_norm) = 85.69
Source: Open LLM Leaderboard

MMLU (5 - Shot)

Task: Text Generation
Dataset: MMLU (5 - Shot) (cais/mmlu with config all, test split, 5 few - shot)
Metric: Accuracy (acc) = 62.65
Source: Open LLM Leaderboard

TruthfulQA (0 - shot)

Task: Text Generation
Dataset: TruthfulQA (0 - shot) (truthful_qa with config multiple_choice, validation split, 0 few - shot)
Metric: mc2 = 66.8
Source: Open LLM Leaderboard

Winogrande (5 - shot)

Task: Text Generation
Dataset: Winogrande (5 - shot) (winogrande with config winogrande_xl, validation split, 5 few - shot)
Metric: Accuracy (acc) = 77.35
Source: Open LLM Leaderboard

GSM8k (5 - shot)

Task: Text Generation
Dataset: GSM8k (5 - shot) (gsm8k with config main, test split, 5 few - shot)
Metric: Accuracy (acc) = 51.4
Source: Open LLM Leaderboard

Overall Metrics

Open LLM Leaderboard Evaluation Results Detailed results can be found here

Metric	Value
Avg.	68.47
AI2 Reasoning Challenge (25 - Shot)	66.89
HellaSwag (10 - Shot)	85.69
MMLU (5 - Shot)	62.65
TruthfulQA (0 - shot)	66.80
Winogrande (5 - shot)	77.35
GSM8k (5 - shot)	51.40

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご