Magnum V2 123b

🚀 Magnum-123b-v1 Model Introduction

This is the sixth model in a series aiming to replicate the prose quality of Claude 3 models, offering multilingual text generation capabilities.

🚀 Quick Start

This model is fine - tuned on top of Mistral - Large - Instruct - 2407. It supports multiple languages including English, French, German, Spanish, Italian, Portuguese, Russian, Chinese, and Japanese.

✨ Features

• Designed to replicate the prose quality of Claude 3 models (Sonnet and Opus).
• Supports multiple languages for text generation.
• Instruct - tuned with the Mistral formatting.

📦 Installation

No installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

The model has been Instruct tuned with the Mistral formatting. A typical input would look like this:

<s>[INST] SYSTEM MESSAGE\nUSER MESSAGE[/INST] ASSISTANT MESSAGE</s>[INST] USER MESSAGE[/INST]

We also provide SillyTavern presets for Context and Instruct respectively. The Mistral preset included in SillyTavern seems to be misconfigured by default, so we recommend using these as a replacement.

📚 Documentation

Credits

• anthracite - org/Stheno - Data - Filtered
• anthracite - org/kalo - opus - instruct - 22k - no - refusal
• anthracite - org/nopm_claude_writing_fixed

This model has been a team effort, and the credits go to all members of Anthracite.

Training

The training was done for 1.5 epochs. We used 8x AMD Instinct™ MI300X Accelerators for the full - parameter fine - tuning of the model.

In addition to this, we noticed that Mistral Large models seemed much more sensitive to learning rate adjustments than other models:

We hypothesize this is primarily due to the particularly narrow and low variance weight distributions typical of Mistral derived models regardless of their scale.

In the end, due to the costs that would be involved in training another full 2 epochs run ($600) on an even lower rate, we settled on our third attempt: 2e - 6 with an effective batch size of 64. We chose to publish the 1.5 epoch run after manually testing and comparing it.

Also, we notice a correlation between the significance of the 2nd epoch loss drop and the strength of the learning rate, implying 4e - 6 leads to more catastrophic forgetting.

[](https://github.com/OpenAccess - AI - Collective/axolotl)

Safety

...

📄 License

• License Name: mrl
• License Link: https://mistral.ai/licenses/MRL - 0.1.md

📋 Information Table