Legion-V2.2-LLaMa-70B Open-source Pretrained Language Model - Integrating Multiple Models to Meet Diverse Text Requirements

Home

Legion V2.2 LLaMa 70B

Developed by TareksTesting

A pre-trained language model merged using the DARE TIES method, based on L-BASE-V1 and fusing multiple MERGE models

Large Language Model

Transformers

#DARE-TIES Merge #Multi-model Fusion #Parameter Weighted Mixing

Downloads 24

Release Time : 3/24/2025

Model Overview

This model is the result of merging multiple pre-trained language models using the mergekit tool, employing the DARE TIES merging method to combine the strengths of each model and enhance overall performance.

Model Features

DARE TIES Merging Method

Utilizes advanced DARE TIES model merging technology to effectively integrate parameters from multiple models

Modular Weight Allocation

Sets separate weight parameters for different modules such as self_attn and mlp

Parameter Density Control

Controls the retention ratio of model parameters through the density parameter

Model Capabilities

Text Generation

Language Understanding

Use Cases

Natural Language Processing

Text Generation

Can be used to generate coherent and fluent text content

Question Answering Systems

Applied in building question-answering systems to respond to user queries

🚀 Merge of Pre-trained Language Models

This project is a merge of pre-trained language models, offering enhanced performance and capabilities through the combination of multiple models.

🚀 Quick Start

This is a merge of pre-trained language models created using mergekit.

📚 Documentation

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using TareksLab/L-BASE-V1 as a base.

Models Merged

The following models were included in the merge:

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: TareksLab/L2-MERGE4
    parameters:
      weight: 
      - filter: self_attn
        value: [0.3, 0.1, 0.2]
      - filter: mlp
        value: [0.4, 0.2, 0.1]
      - value: 0.2
      density: 0.7
      lambda: 1.05
  - model: TareksLab/L2-MERGE2a
    parameters:
      weight: 
      - filter: self_attn
        value: [0.2, 0.1, 0.3]
      - filter: mlp
        value: [0.3, 0.1, 0.2]
      - value: 0.2
      density: 0.65
      lambda: 1.05
  - model: TareksLab/L2-MERGE3
    parameters:
      weight: 
      - filter: self_attn
        value: [0.1, 0.3, 0.1]
      - filter: mlp
        value: [0.2, 0.3, 0.1]
      - value: 0.2
      density: 0.6
      lambda: 1.05
  - model: TareksLab/L2-MERGE1
    parameters:
      weight: 
      - filter: self_attn
        value: [0.2, 0.2, 0.1]
      - filter: mlp
        value: [0.1, 0.2, 0.2]
      - value: 0.2
      density: 0.6
      lambda: 1
  - model: TareksLab/L-BASE-V1
    parameters:
      weight: 
      - filter: self_attn
        value: [0.1, 0.3, 0.3]
      - filter: mlp
        value: [0.1, 0.2, 0.4]
      - value: 0.2
      density: 0.55
      lambda: 1
base_model: TareksLab/L-BASE-V1
merge_method: dare_ties
parameters:
  normalize: false
  pad_to_multiple_of: 4
tokenizer:
  source: base
chat_template: llama3
dtype: bfloat16

Information Table

Property	Details
Base Models	TareksLab/L2-MERGE2a, TareksLab/L2-MERGE4, TareksLab/L-BASE-V1, TareksLab/L2-MERGE3, TareksLab/L2-MERGE1
Library Name	transformers
Tags	mergekit, merge

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご