M7-7b Open-source Fusion Model - Combine the Advantages of Multiple Models to Improve Performance for Free

M7 7b

Developed by liminerity

M7-7b is an experimental project that uses the mergekit tool to fuse multiple models with a 7B parameter scale, aiming to integrate the advantages of different models to improve performance.

Large Language Model

Transformers

Open Source License:Apache-2.0 #Multi-model fusion #SLERP mixing #7B parameter scale

Downloads 8,909

Release Time : 3/7/2024

Model Overview

This model uses the slerp method to fuse multiple models with 7B parameters, including liminerity/merge4 and liminerity/merge2, etc., to achieve more powerful performance by integrating the advantages of different models.

Model Features

Multi-model fusion

Fuse multiple 7B parameter models through the slerp method to integrate the advantages of different models

Hierarchical parameter configuration

Use different fusion parameter configurations for the self_attn and mlp layers

Experimental architecture

Adopt an innovative model fusion strategy to explore the possibility of performance improvement

Model Capabilities

Text generation

Language understanding

Multi-task processing

Use Cases

Natural language processing

Text generation

Generate coherent and logical text content

Question-answering system

Build a knowledge question-answering application

🚀 M7-7b

M7-7b, also known as Multiverse-Experiment-slerp-7b, is a merged model created by combining multiple models using mergekit. This approach allows for leveraging the strengths of different models to potentially achieve better performance.

✨ Features

Model Merging: It merges the following models:
- liminerity/merge4
- liminerity/merge2

📚 Documentation

🧾 Configuration

The configuration for merging these models is as follows:

slices:
  - sources:
      - model: ammarali32/multi_verse_model
        layer_range: [0, 32]
      - model: MSL7/INEX12-7b
        layer_range: [0, 32]
merge_method: slerp
base_model: ammarali32/multi_verse_model
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5
dtype: bfloat16

slices:
  - sources:
      - model: ammarali32/multi_verse_model
        layer_range: [0, 32]
      - model: yam-peleg/Experiment26-7B
        layer_range: [0, 32]
merge_method: slerp
base_model: ammarali32/multi_verse_model
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5
dtype: bfloat16

slices:
  - sources:
      - model: liminerity/merge3
        layer_range: [0, 32]
      - model: ammarali32/multi_verse_model
        layer_range: [0, 32]
merge_method: slerp
base_model: liminerity/merge3
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5
dtype: bfloat16

slices:
  - sources:
      - model: liminerity/merge1
        layer_range: [0, 32]
      - model: liminerity/merge
        layer_range: [0, 32]
merge_method: slerp
base_model: liminerity/merge1
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5
dtype: bfloat16

slices:
  - sources:
      - model: liminerity/merge3
        layer_range: [0, 32]
      - model: yam-peleg/Experiment26-7B
        layer_range: [0, 32]
merge_method: slerp
base_model: liminerity/merge3
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5
dtype: bfloat16

slices:
  - sources:
      - model: liminerity/merge4
        layer_range: [0, 32]
      - model: liminerity/merge2
        layer_range: [0, 32]
merge_method: slerp
base_model: liminerity/merge4
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5
dtype: bfloat16

📄 License

This project is licensed under the Apache-2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご