Fimbulvetr-Kuro-Lotus-10.7B Open-source Hybrid Model - Excellent Text Generation Results and Free to Use!

Fimbulvetr Kuro Lotus 10.7B

Developed by saishf

This is a hybrid model created through SLERP (Spherical Linear Interpolation) fusion of two 10.7B parameter models, combining the strengths of Fimbulvetr and Kuro-Lotus, demonstrating excellent performance in text generation tasks.

Large Language Model

Transformers

#Long-context dialogue optimization #Enhanced role-playing #SLERP fusion model

Downloads 57

Release Time : 2/13/2024

Model Overview

The model integrates features from both Fimbulvetr and Kuro-Lotus 10.7B parameter models, specializing in text generation tasks, particularly suited for long-context dialogues and creative writing.

Model Features

Long-context retention capability

Maintains coherence in dialogues with contexts up to 8K tokens

Creative text generation

Produces highly creative outputs when using the 'Universal Light' preset template

Flexible preset adaptation

Supports multiple preset templates (e.g., alpaca, alpaca roleplay, default, etc.)

Model Capabilities

Long text generation

Dialogue systems

Creative writing

Reasoning tasks

Use Cases

Dialogue systems

Role-playing dialogues

Used for role-playing dialogues on platforms like Silly Tavern

Produces coherent dialogues without requiring highly detailed character cards

Creative writing

Story generation

Creative story writing using the 'Universal Light' preset

Generates imaginative narrative content

🚀 Fimbulvetr-Kuro-Lotus-10.7B

This model is a merge of two pre - trained language models, offering high - quality text generation capabilities and achieving good results in multiple evaluation tasks.

🚀 Quick Start

This model is a merge of personal favorite models. It provides Kuro - Lotus like results without the need for a highly detailed character card and maintains coherence in up to 8K context. You can use presets like "Universal Light" in silly tavern for text generation.

✨ Features

Model Merge: It is a merge of pre - trained language models using the mergekit tool.
Good Performance: Achieves good scores in various text - generation evaluation tasks such as AI2 Reasoning Challenge, HellaSwag, MMLU, etc.
Context Coherence: Stays coherent when handling up to 8K context.

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

No code examples are provided in the original document, so this section is skipped.

📚 Documentation

Merge Details

Merge Method

This model was merged using the SLERP merge method.

Models Merged

The following models were included in the merge:

Configuration

The following YAML configuration was used to produce this model:

slices:
  - sources:
      - model: saishf/Kuro-Lotus-10.7B
        layer_range: [0, 48]
      - model: Sao10K/Fimbulvetr-10.7B-v1
        layer_range: [0, 48]
merge_method: slerp
base_model: saishf/Kuro-Lotus-10.7B
parameters:
  t:
    - filter: self_attn
      value: [0.6, 0.7, 0.8, 0.9, 1]
    - filter: mlp
      value: [0.4, 0.3, 0.2, 0.1, 0]
    - value: 0.5
dtype: bfloat16

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Property	Details
Model Type	Fimbulvetr-Kuro-Lotus-10.7B
Training Data	Not provided

Metric	Value
Avg.	72.73
AI2 Reasoning Challenge (25 - Shot)	69.54
HellaSwag (10 - Shot)	87.87
MMLU (5 - Shot)	66.99
TruthfulQA (0 - shot)	60.95
Winogrande (5 - shot)	84.14
GSM8k (5 - shot)	66.87

🔧 Technical Details

No technical details over 50 words are provided in the original document, so this section is skipped.

📄 License

This model is licensed under the CC - BY - NC - 4.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご