Ret CLIP ViT L 14

🚀 Model Card for ReT - Multimodal Document Retrieval

ReT is a cutting - edge solution for multimodal document retrieval, supporting both multimodal queries and documents. Unlike traditional methods relying solely on final - layer features of vision - and - language backbones, ReT uses a Transformer - based recurrent cell. It leverages multi - level representations from different layers of visual and textual backbones. The model features sigmoidal gates, inspired by LSTM design, to selectively control information flow between layers and modalities. ReT processes multimodal queries and documents independently, generating latent tokens for fine - grained late interaction similarity computation. It is designed to handle images and text in both queries and documents, and has been trained and evaluated on a customized version of the challenging M2KR benchmark.

🚀 Quick Start

Prerequisites

Follow the instructions on the repository to install the required environment.

Usage Example

from src.models import RetrieverModel, RetModel
import torch

device = 'cuda' if torch.cuda.is_available() else 'cpu'
retriever = RetrieverModel.from_pretrained('aimagelab/ReT-CLIP-ViT-L-14', device_map=device)

# QUERY
ret: RetModel = retriever.get_query_model()
ret.init_tokenizer_and_image_processor()
q_txt = "Retrieve documents that provide an answer to the question alongside the image: What is the content of the image?"
q_img = 'assets/model.png'

ret_feats = ret.get_ret_features([[q_txt, q_img]])
print(ret_feats.shape)  # torch.Size([1, 32, 128])


# PASSAGE
ret: RetModel = retriever.get_passage_model()
ret.init_tokenizer_and_image_processor()

p_txt = """The image shows a diagram of what appears to be a neural network architecture using a fine-grained loss approach for multimodal learning.
The architecture has two parallel processing streams labeled "ReTQ" (left side, in purple) and "ReTD" (right side, in blue).
Each side has: ..."""
p_img = ''

ret_feats = ret.get_ret_features([[p_txt, p_img]])
print(ret_feats.shape)  # torch.Size([1, 32, 128])

✨ Features

• Multimodal Support: ReT can handle both images and text in queries and documents.
• Multi - Level Representation: It leverages multi - level features from different layers of visual and textual backbones.
• Selective Information Flow: Sigmoidal gates inspired by LSTM control information flow between layers and modalities.

📦 Installation

Refer to the repository for detailed installation instructions.

📚 Documentation

Model Sources

• Repository: https://github.com/aimagelab/ReT
• Paper: Recurrence - Enhanced Vision - and - Language Transformers for Robust Multimodal Document Retrieval (CVPR 2025)

Model Details

📄 License

This project is licensed under the Apache 2.0 license.

📄 Citation

BibTeX:

@inproceedings{caffagni2025recurrence,
  title={{Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrieval}},
  author={Caffagni, Davide and Sarto, Sara and Cornia, Marcella and Baraldi, Lorenzo and Cucchiara, Rita},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2025}
}