Discolm Mfto 7b German V0.1

🚀 Merge

This is an experimental merge of pre-trained language models created using mergekit, aiming to combine the strengths of different models for better German language generation, especially in role - play scenarios.

🚀 Quick Start

This README provides details about an experimental merge of pre - trained language models. It explains the merge method, the models involved, and the configuration used.

✨ Features

• Multilingual Support: Supports both German (de) and English (en).
• Targeted for German Role - play: Aims to create a better model for German role - play scenarios.
• Combined Strengths: Merges models known for their performance in German language generation.

📦 Installation

No installation steps are provided in the original document.

💻 Usage Examples

No code examples are provided in the original document.

📚 Documentation

🔍 Merge Details

📋 Merge Method

This model was merged using the SLERP merge method.

📌 Models Merged

The following models were included in the merge:

• [OpenPipe/mistral - ft - optimized - 1227](https://huggingface.co/OpenPipe/mistral - ft - optimized - 1227)
• DiscoResearch/DiscoLM_German_7b_v1

❓ Why these two models?

As far as I know, both used models are the two best models for German language generation.

As of 01/21/2024, DiscoLM German 7B is by far the best German model, making far fewer grammatical errors and generally producing good - sounding German. However, it is fine - tuned on Mistral V0.2 or even V0.1.

Mistral FT Optimized 1227 performs much better in German than Mistral 7B V0.2 and other German fine - tuning models that make grammar errors in almost every sentence. Nevertheless, it still lags behind DiscoLM German 7B and generates less well - formed German sentences.

The ulterior motive was to combine these two models to obtain an even better German model, especially for German role - play.

⚙️ Configuration

The following YAML configuration was used to produce this model:

slices:
  - sources:
      - model: OpenPipe/mistral-ft-optimized-1227
        layer_range: [0, 32]
      - model: DiscoResearch/DiscoLM_German_7b_v1
        layer_range: [0, 32]
merge_method: slerp
base_model: OpenPipe/mistral-ft-optimized-1227
parameters:
  t:
    - value: [0.5, 0.9]
dtype: bfloat16

These settings are from the model [oshizo/japanese - e5 - mistral - 7b_slerp](https://huggingface.co/oshizo/japanese - e5 - mistral - 7b_slerp).

🔧 Technical Details

The document provides details about the merge method (SLERP) and the specific models used in the merge, as well as the YAML configuration for the merge process.

📄 License

No license information is provided in the original document.