Llama2 22b

🚀 Llama 2 13b with Additional Attention Heads

This is a modified version of Llama 2 13b, with some additional attention heads from the original - flavor Llama 33b integrated. It aims to serve as a base for further tuning with potentially greater learning capacity than the standard 13b model.

🚀 Quick Start

This model is not intended for direct use. It is designed to be a foundation for further fine - tuning.

✨ Features

• This model combines the structure of Llama 2 13b with additional attention heads from Llama 33b.
• It has been fine - tuned on approximately 10M tokens from RedPajama to better adapt to the transplanted components.

📚 Documentation

Model Information

Model Explanation

This is Llama 2 13b with some additional attention heads from original - flavor Llama 33b frankensteined on. It was fine - tuned on ~10M tokens from RedPajama to settle in the transplants a little. It's not meant for direct use but as a base for further tuning, hopefully with a greater capacity for learning than 13b.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here