Stripedhyena Hessian 7B

🚀 StripedHyena-Hessian-7B (SH 7B)

StripedHyena-Hessian-7B (SH 7B) is a base model released by Together Research. It is based on the StripedHyena architecture, which is the first alternative model competitive with the best open - source Transformers of similar sizes in short and long - context evaluations.

🚀 Quick Start

• Read more about the model in our blog.
• Play with the model on our playground.
• Explore the details of our standalone implementation, and our related research: 1, 2, 3.

✨ Features

One of the focus areas at Together Research is new architectures for long context, improved training, and inference performance over the Transformer architecture. The StripedHyena models spin out of a research program from the team and academic collaborators, with roots in signal processing - inspired sequence models.

Model Architecture

StripedHyena is a hybrid architecture composed of multi - head, grouped - query attention and gated convolutions arranged in Hyena blocks, different from traditional decoder - only Transformers.

• It enables constant memory decoding in Hyena blocks via representation of convolutions as state - space models (modal or canonical form), or as truncated filters.
• It has low latency, faster decoding, and higher throughput than Transformers.
• It improves training and inference - optimal scaling laws compared to optimized Transformer architectures such as Llama - 2.
• It is trained on sequences of up to 32k, allowing it to process longer prompts.

📚 Documentation

Note

To use StripedHyena outside of the playground, you will need to install custom kernels. Please follow the instructions from the standalone repository.

StripedHyena is a mixed precision model. Make sure to keep your poles and residues in float32 precision, especially for longer prompts or training.

⚠️ Important Note

To use StripedHyena outside of the playground, custom kernels installation is required. Follow the instructions from the standalone repository.

💡 Usage Tip

Since StripedHyena is a mixed precision model, keep your poles and residues in float32 precision, especially for longer prompts or training.

📄 License

This project is licensed under the Apache - 2.0 license.

🔧 Technical Details

If you have found the pretrained models or architecture useful for your research or application, consider citing:

@software{stripedhyena,
  title        = {{StripedHyena: Moving Beyond Transformers with Hybrid Signal Processing Models}},
  author       = { Poli, Michael and Wang, Jue and Massaroli, Stefano and Quesnelle, Jeffrey and Carlow, Ryan and Nguyen, Eric and Thomas, Armin},
  month        = 12,
  year         = 2023,
  url          = { https://github.com/togethercomputer/stripedhyena },
  doi          = { 10.57967/hf/1595 },
}