Fireblossom-32K-7B Open-Source Language Model - Supports 32K Context, Balancing Creativity and Reasoning

Fireblossom 32K 7B

Developed by grimjim

A 7B-parameter language model merged from Mistral 7B v0.1, combining multiple fine-tuned models via task arithmetic, supporting 32K context length, balancing creativity and reasoning

Large Language Model

Transformers

#Long Context Processing #Creative Text Generation #Multi-Model Fusion

Downloads 21

Release Time : 4/14/2024

Model Overview

This is a 7B-parameter language model merged using mergekit, based on Mistral 7B v0.1, combining multiple fine-tuned models with strong narrative role-playing and reasoning capabilities, aiming to provide diverse text outputs

Model Features

Extended Context Length

Supports 32K context length by adjusting rope theta parameters

Diverse Outputs

Balances creativity and accuracy to provide more diverse text generation

Multi-Model Advantage Fusion

Integrates multiple excellent fine-tuned models with strong narrative role-playing and reasoning capabilities

Model Capabilities

Long Text Generation

Role-Playing Dialogue

Reasoning Task Processing

Instruction Following

Use Cases

Creative Writing

Story Creation

Generate long stories with coherent plots

Can maintain coherent narratives within 32K context

Dialogue Systems

Role-Playing Chat

Engage in role-playing style dialogue interactions

Exhibits vivid character personalities and dialogues

🚀 Fireblossom-32K-7B

This is a merged pre - trained language model created using mergekit, aiming to provide more diverse outputs.

🚀 Quick Start

This model is a merge of pre - trained language models using mergekit. For the merge, Mistral 7B v0.1 was used as the literal base model for task arithmetic merger. After adjusting rope theta from 10K to 100K, it can support at least 16K context length. With the original base model, the merged models are mathematically equivalent to LoRA adapters, and the original 32K context claimed by Mistral 7B v0.1 is retained.

The goal was to create a merge model with more diverse outputs, which sacrifices some accuracy for creativity. A model strong at narrative role - play (cgato's work) and three models good at reasoning (fine - tunes by HuggingFaceH4 and SanjiWatsuki) were selected. The result seems to be good at following card instructions.

Sampler settings: It was lightly tested with temperature = 0.7 and minP = 0.01. Increase the temperature for greater creativity.

Prompts: Alpaca format is natively supported, although ChatML was used during testing.

Download options:

📚 Documentation

Merge Details

Merge Method

This model was merged using the task arithmetic merge method with mistralai/Mistral-7B-v0.1 as the base.

Models Merged

The following models were included in the merge:

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: mistralai/Mistral-7B-v0.1
    # no parameters necessary for base model
  - model: SanjiWatsuki/Kunoichi-DPO-v2-7B
    parameters:
      weight: 0.45
  - model: cgato/TheSpice-7b-v0.1.1
    parameters:
      weight: 0.05
  - model: HuggingFaceH4/zephyr-7b-beta
    parameters:
      weight: 0.05
  - model: SanjiWatsuki/Kunoichi-7B
    parameters:
      weight: 0.45
merge_method: task_arithmetic
base_model: mistralai/Mistral-7B-v0.1
dtype: float16

📄 License

This project is licensed under the cc - by - nc - 4.0 license.

Property	Details
Base Model	HuggingFaceH4/zephyr-7b-beta, cgato/TheSpice-7b-v0.1.1, SanjiWatsuki/Kunoichi-DPO-v2-7B, SanjiWatsuki/Kunoichi-7B, mistralai/Mistral-7B-v0.1
Library Name	transformers
Tags	mergekit, merge
License	cc-by-nc-4.0
Pipeline Tag	text-generation

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご