13B Thorns L2

🚀 13B-Thorns [An Instruct Based LLaMAv2-13B Ensemble Merge | Alpaca Format]

13B-Thorns is an ensemble merge model based on LLaMAv2-13B, following the Alpaca format, aiming to combine the strengths of multiple models.

⚠️ Important Note

This model is uncensored and has not been fully tested for toxicity. It is a research artifact intended for responsible use. It may generate offensive and misleading content. Do not treat the language synthesized by this research artifact as advice or as factual in any domain. CalderaAI strictly does not condone the use of this release outside the domain of research or entertainment.

✨ Features

Composition

13B-Thorns-l2 utilizes a new merge method called Spherical Linear Interpolation. By merging data as a spherical vector store concept, a combined pair of models have a smoother transition between feature spaces that are characteristic of each model, resulting in a more coherent fusion of both model's unique strengths.

Our implementation of Spherical Linear Interpolation for LLM merging

https://github.com/Digitous/LLM-SLERP-Merge

Thorns' Design Concept

Thorns' design is based on the concept of purposed segmentation, which includes two segments:

--Logic Segment (MK1)

Fine-Tuned parent models were hand - selected and reviewed for datasets, performance, least restrictive censorship, and community perception of coherence and utility. Ultimately, four models were decided to be merged in pairs of two, then combine those offspring for a quad - merged logic cluster. All four models were merged using the SLERP method.

--Creativity and Imagination Segment (MK1)

The first approach was flawed. We initially thought the creativity and imagination segment could be a single model, like KoboldAI's Holodeck model. However, we got carried away with adding LoRAs. We added the LIMA RP PEFT to Holodeck and then two more LoRAs (Kimiko and Janine). When we SLERP merged this version of the imagination segment to the logic segment, the result was a ranting mess.

Finalized Design (TL;DR)

After re - working our approach, we drafted Thorns MK2. The final merge is as follows:

-Model Merge Ensemble Key-

{} = SLERP Merge | [] = PEFT Merge | () = Composite Model

({({NousHermes+Chronos}[Kimiko])+({Platupus+AiroborosM2.0}[Janine])}{Holodeck[LIMA RP]})

Findings

• Strategically fusing LoRAs to models that stand to gain the most from them and then merging the result into the ensemble is exceptionally effective.
• Stacking the exact same LoRAs onto one model then merging that into the ensemble results in noisy garbage.

📄 License

All models and adapters used are LLaMAv2 - 13B.

Models

• Nous - Hermes
• Chronos
• Platypus
• Airoboros
• Holodeck

Adapters

• Kimiko
• Janine
• LIMA RP

Also, thanks to Meta for LLaMAv2 and allowing the research community to benefit from their work. Each model and LoRA was hand - picked for its contribution to this ensemble. Thanks to all the developers for their great work.