đ CosmicBun-8B
CosmicBun-8B is a merged pre - trained language model, leveraging the power of multiple base models to achieve excellent performance in text - generation tasks.
đ Documentation
Model Overview
This is a merge of pre - trained language models created using mergekit.
Merge Method
This model was merged using the DARE TIES merge method using [Locutusque/llama - 3 - neural - chat - v1 - 8b](https://huggingface.co/Locutusque/llama - 3 - neural - chat - v1 - 8b) as a base.
Models Merged
The following models were included in the merge:
- [cognitivecomputations/dolphin - 2.9 - llama3 - 8b](https://huggingface.co/cognitivecomputations/dolphin - 2.9 - llama3 - 8b)
- [Weyaxi/Einstein - v6.1 - Llama3 - 8B](https://huggingface.co/Weyaxi/Einstein - v6.1 - Llama3 - 8B)
Configuration
The following YAML configuration was used to produce this model:
base_model: Locutusque/llama-3-neural-chat-v1-8b
dtype: bfloat16
merge_method: dare_ties
parameters:
int8_mask: 1.0
normalize: 0.0
slices:
- sources:
- layer_range: [0, 4]
model: cognitivecomputations/dolphin-2.9-llama3-8b
parameters:
density: 1.0
weight: 0.6
- layer_range: [0, 4]
model: Weyaxi/Einstein-v6.1-Llama3-8B
parameters:
density: 0.6
weight: 0.5
- layer_range: [0, 4]
model: Locutusque/llama-3-neural-chat-v1-8b
parameters:
density: 1.0
weight: 0.5
- sources:
- layer_range: [4, 8]
model: cognitivecomputations/dolphin-2.9-llama3-8b
parameters:
density: 0.8
weight: 0.1
- layer_range: [4, 8]
model: Weyaxi/Einstein-v6.1-Llama3-8B
parameters:
density: 1.0
weight: 0.2
- layer_range: [4, 8]
model: Locutusque/llama-3-neural-chat-v1-8b
parameters:
density: 1.0
weight: 0.7
- sources:
- layer_range: [8, 12]
model: cognitivecomputations/dolphin-2.9-llama3-8b
parameters:
density: 0.7
weight: 0.1
- layer_range: [8, 12]
model: Weyaxi/Einstein-v6.1-Llama3-8B
parameters:
density: 0.7
weight: 0.2
- layer_range: [8, 12]
model: Locutusque/llama-3-neural-chat-v1-8b
parameters:
density: 0.7
weight: 0.6
- sources:
- layer_range: [12, 16]
model: cognitivecomputations/dolphin-2.9-llama3-8b
parameters:
density: 0.9
weight: 0.2
- layer_range: [12, 16]
model: Weyaxi/Einstein-v6.1-Llama3-8B
parameters:
density: 0.6
weight: 0.6
- layer_range: [12, 16]
model: Locutusque/llama-3-neural-chat-v1-8b
parameters:
density: 0.7
weight: 0.3
- sources:
- layer_range: [16, 20]
model: cognitivecomputations/dolphin-2.9-llama3-8b
parameters:
density: 1.0
weight: 0.2
- layer_range: [16, 20]
model: Weyaxi/Einstein-v6.1-Llama3-8B
parameters:
density: 1.0
weight: 0.2
- layer_range: [16, 20]
model: Locutusque/llama-3-neural-chat-v1-8b
parameters:
density: 0.9
weight: 0.4
- sources:
- layer_range: [20, 24]
model: cognitivecomputations/dolphin-2.9-llama3-8b
parameters:
density: 0.7
weight: 0.2
- layer_range: [20, 24]
model: Weyaxi/Einstein-v6.1-Llama3-8B
parameters:
density: 0.9
weight: 0.3
- layer_range: [20, 24]
model: Locutusque/llama-3-neural-chat-v1-8b
parameters:
density: 1.0
weight: 0.4
- sources:
- layer_range: [24, 28]
model: cognitivecomputations/dolphin-2.9-llama3-8b
parameters:
density: 1.0
weight: 0.4
- layer_range: [24, 28]
model: Weyaxi/Einstein-v6.1-Llama3-8B
parameters:
density: 0.8
weight: 0.2
- layer_range: [24, 28]
model: Locutusque/llama-3-neural-chat-v1-8b
parameters:
density: 0.9
weight: 0.4
- sources:
- layer_range: [28, 32]
model: cognitivecomputations/dolphin-2.9-llama3-8b
parameters:
density: 1.0
weight: 0.3
- layer_range: [28, 32]
model: Weyaxi/Einstein-v6.1-Llama3-8B
parameters:
density: 0.9
weight: 0.2
- layer_range: [28, 32]
model: Locutusque/llama-3-neural-chat-v1-8b
parameters:
density: 1.0
weight: 0.3
Detailed results can be found here
Metric |
Value |
Avg. |
68.81 |
AI2 Reasoning Challenge (25 - Shot) |
61.86 |
HellaSwag (10 - Shot) |
84.29 |
MMLU (5 - Shot) |
65.53 |
TruthfulQA (0 - shot) |
54.08 |
Winogrande (5 - shot) |
78.85 |
GSM8k (5 - shot) |
68.23 |
đ License
This project is licensed under the MIT license.