PE Lang G14 448

🚀 Perception Encoder

Perception Encoder (PE) is a state - of - the - art encoder for image and video understanding. It's trained via simple vision - language learning and offers excellent performance across various vision tasks.

🚀 Quick Start

For model loading code, we provide it in this GitHub repository. You can find more details there.

✨ Features

• State - of - the - art Performance: Perception Encoder (PE) is a family of large - scale vision encoder models. It outperforms all existing models on classification and retrieval tasks, and internally produces strong, general features for downstream tasks.
• Versatile for Language Modeling: PE lang takes strong language performance from the intermediate layers of PE core and further aligns for language modeling. It's versatile for any multimodal language modeling use case, including using different language model decoders (e.g., Llama / Qwen) and different eval settings (e.g., native res / tiling). It performs well on OCR and document tasks.

📚 Documentation

Model Details

[📃 Tech Report] [📂 Github]

Perception Encoder (PE) was introduced in "Perception Encoder: The best visual embeddings are not at the output of the network".

Model Developer: Meta

Model Overview: Perception Encoder (PE) uses a robust contrastive pretraining recipe and finetuning on synthetically aligned videos. It unlocks the ability for large - scale contrastive pretraining to transfer to downstream tasks with alignment tuning.

Perception Encoder: Language

PE lang further aligns for language modeling following PLM. We release two PE Lang checkpoints, L14 - 448 and G14 - 448.

Here are their results in our benchmark setting with frozen encoder with 2.6M SFT datamix, using 448px only (i.e., with no tiling) and Llama 3.1 8B as the decoder:

Here is a sample of the performance obtainable by using PE Core G aligned further with [PLM - 8B](https://huggingface.co/facebook/Perception - LM - 8B) (stage 3) using 36 + 1 image tiles / 32 video frames with Llama 3.1 8B as the decoder:

* The PE - Core - G14 - 448 checkpoint was further trained using tiling. We will release the tiling aligned checkpoint soon.

See the paper for full performance evaluations and fair comparisons to other models.

📄 License

This project is licensed under the Apache - 2.0 license.

📚 Citation

If you find our code useful for your research, please consider citing:

@article{bolya2025PerceptionEncoder,
  title={Perception Encoder: The best visual embeddings are not at the output of the network},
  author={Daniel Bolya and Po - Yao Huang and Peize Sun and Jang Hyun Cho and Andrea Madotto and Chen Wei and Tengyu Ma and Jiale Zhi and Jathushan Rajasegaran and Hanoona Rasheed and Junke Wang and Marco Monteiro and Hu Xu and Shiyu Dong and Nikhila Ravi and Daniel Li and Piotr Doll{\'a}r and Christoph Feichtenhofer},
  journal={arXiv},
  year={2025}
}

@article{cho2025PerceptionLM,
  title={PerceptionLM: Open - Access Data and Models for Detailed Visual Understanding},
  author={Jang Hyun Cho and Andrea Madotto and Effrosyni Mavroudi and Triantafyllos Afouras and Tushar Nagarajan and Muhammad Maaz and Yale Song and Tengyu Ma and Shuming Hu and Hanoona Rasheed and Peize Sun and Po - Yao Huang and Daniel Bolya and Suyog Jain and Miguel Martin and Huiyu Wang and Nikhila Ravi and Shashank Jain and Temmy Stark and Shane Moon and Babak Damavandi and Vivian Lee and Andrew Westbury and Salman Khan and Philipp Kr\"{a}henb\"{u}hl and Piotr Doll{\'a}r and Lorenzo Torresani and Kristen Grauman and Christoph Feichtenhofer},
  journal={arXiv},
  year={2025}
}