đ Language Model Merge Project
This project focuses on merging pre - trained language models, aiming to combine the strengths of multiple models to create a more powerful and versatile language model.
đ Quick Start
The core of this project is to use the mergekit library to merge pre - trained language models. The main steps involve specifying the base model and the models to be merged, and then using a specific merge method to generate the new model.
⨠Features
- Multiple Base Models: Supports multiple pre - trained language models as bases, including
LatitudeGames/Wayfarer - 12B, yamatazen/EtherealAurora - 12B - v2, etc.
- Advanced Merge Method: Utilizes the Model Stock merge method to ensure high - quality model merging.
- Visual Representation: Comes with a GIF to vividly demonstrate the merge process.
đĻ Installation
The original README does not provide specific installation steps, so this section is skipped.
đģ Usage Examples
The original README does not provide code examples, so this section is skipped.
đ Documentation
Merge Details
Merge Method
This model was merged using the Model Stock merge method, with [yamatazen/EtherealAurora - 12B - v2](https://huggingface.co/yamatazen/EtherealAurora - 12B - v2) as the base.
Models Merged
The following models were included in the merge:
- [LatitudeGames/Wayfarer - 12B](https://huggingface.co/LatitudeGames/Wayfarer - 12B)
- [TheDrummer/Rocinante - 12B - v1.1](https://huggingface.co/TheDrummer/Rocinante - 12B - v1.1)
- [nbeerbower/Lyra4 - Gutenberg - 12B](https://huggingface.co/nbeerbower/Lyra4 - Gutenberg - 12B)
- [MarinaraSpaghetti/NemoMix - Unleashed - 12B](https://huggingface.co/MarinaraSpaghetti/NemoMix - Unleashed - 12B)
- [cognitivecomputations/dolphin - 2.9.3 - mistral - nemo - 12b](https://huggingface.co/cognitivecomputations/dolphin - 2.9.3 - mistral - nemo - 12b)
- [nothingiisreal/MN - 12B - Celeste - V1.9](https://huggingface.co/nothingiisreal/MN - 12B - Celeste - V1.9)
- [anthracite - org/magnum - v2 - 12b](https://huggingface.co/anthracite - org/magnum - v2 - 12b)
Configuration
The following YAML configuration was used to produce this model:
models:
- model: LatitudeGames/Wayfarer-12B
- model: MarinaraSpaghetti/NemoMix-Unleashed-12B
- model: nothingiisreal/MN-12B-Celeste-V1.9
- model: TheDrummer/Rocinante-12B-v1.1
- model: anthracite-org/magnum-v2-12b
- model: nbeerbower/Lyra4-Gutenberg-12B
- model: cognitivecomputations/dolphin-2.9.3-mistral-nemo-12b
merge_method: model_stock
base_model: yamatazen/EtherealAurora-12B-v2
normalize: false
int8_mask: true
dtype: bfloat16
đ§ Technical Details
The original README does not provide sufficient technical details, so this section is skipped.
đ License
The original README does not provide license information, so this section is skipped.
Visual Representation
This is a merge of pre - trained language models created using mergekit.
Information Table
| Property |
Details |
| Base Models |
LatitudeGames/Wayfarer - 12B, yamatazen/EtherealAurora - 12B - v2, TheDrummer/Rocinante - 12B - v1.1, nbeerbower/Lyra4 - Gutenberg - 12B, MarinaraSpaghetti/NemoMix - Unleashed - 12B, cognitivecomputations/dolphin - 2.9.3 - mistral - nemo - 12b, nothingiisreal/MN - 12B - Celeste - V1.9, anthracite - org/magnum - v2 - 12b |
| Library Name |
transformers |
| Tags |
mergekit, merge |
| Merge Method |
Model Stock |
| Base Model for Merge |
yamatazen/EtherealAurora - 12B - v2 |
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
