Tito 7B Slerp

🚀 Tito-7B-slerp

Tito-7B-slerp is a merged model that combines the strengths of multiple models. It addresses the need for high - performance text generation by leveraging the features of its base models. This model is suitable for various NLP tasks, offering improved accuracy and performance in different evaluation benchmarks.

🚀 Quick Start

Tito-7B-slerp can be used for text generation tasks. You can start by integrating it with your existing NLP pipelines. For more details on how to use it, refer to the official documentation of the base models and the mergekit tool.

✨ Features

• Model Merging: Utilizes mergekit to combine gordicaleksa/YugoGPT and mlabonne/AlphaMonarch-7B.
• Slerp Merge Method: Employs the slerp merge method for better model integration.
• Parameter Tuning: Allows for fine - tuning of parameters such as t for different components like self_attn and mlp.

📦 Installation

There is no specific installation information provided in the original README. If you want to use this merged model, you may need to follow the installation steps of mergekit and the base models.

📚 Documentation

🧩 Configuration

slices:
  - sources:
      - model: gordicaleksa/YugoGPT
        layer_range: [0, 32]
      - model: mlabonne/AlphaMonarch-7B
        layer_range: [0, 32]
merge_method: slerp
base_model: mlabonne/AlphaMonarch-7B
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.6
dtype: bfloat16

Results

Evaluations on Serbian LLM eval suite

Evaluations on Serbian LLM eval suite (or rather, performance and knowledge of Serbian):

Here, all benchmarks were done 0 - shot, on the exception of NQ Open and TriviaQA which were done in 5 - shot manner, in order to be comparable to Mistral paper.

Replicating OpenLLM Leaderboard results on Serbian datasets

If we try to replicate OpenLLM Leaderboard results on available Serbian datasets (running an appropriate amount of shots instead of 0), we get:

Note that YugoGPT, Llama3 and SambaLingo are all base models, unlike Tito and Perucac.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

📄 License

This project is licensed under the Apache - 2.0 license.