Hiera Base 224

🚀 Hiera (hiera_base_224)

Hiera is a hierarchical transformer that offers a more efficient alternative to previous hierarchical transformers like ConvNeXT and Swin.

🚀 Quick Start

Hiera is a hierarchical transformer, presenting a far more efficient option compared to previous series of hierarchical transformers such as ConvNeXT and Swin. Vanilla transformer architectures (Dosovitskiy et al. 2020) are highly popular, being simple and scalable architectures that support pretraining strategies like MAE (He et al., 2022).

However, they maintain the same spatial resolution and number of channels throughout the network, leading to inefficient use of parameters in ViTs. This contrasts with prior “hierarchical” or “multi - scale” models (e.g., Krizhevsky et al. (2012); He et al. (2016)), which use fewer channels but higher spatial resolution in the early stages with simpler features, and more channels but lower spatial resolution later in the model with more complex features.

These models are overly complex, adding overhead operations to achieve state - of - the - art accuracy in ImageNet - 1k, thus slowing down the model. Hiera tries to solve this problem by teaching the model spatial biases through MAE training.

📄 License

This project is licensed under the CC BY - NC 4.0 license.