Colqwen2 2b V1.0

🚀 ColQwen2-2B: Visual Retriever based on Qwen2-VL-2B-Instruct with ColBERT strategy

ColQwen2-2B is a visual retriever that uses the ColBERT strategy based on Qwen2-VL-2B-Instruct. It efficiently indexes documents from visual features and generates multi-vector representations of text and images.

✨ Features

• Based on a novel model architecture and training strategy of Vision Language Models (VLMs) to efficiently index documents from visual features.
• An extension of Qwen2-VL-2B that generates ColBERT-style multi-vector representations of text and images.
• Takes dynamic image resolutions in input without resizing and changing aspect ratio, with a maximal resolution set to create at most 1024 image patches.

📦 Installation

Make sure colpali-engine is installed from source or with a version superior to 0.3.4. transformers version must be > 4.46.1.

pip install git+https://github.com/illuin-tech/colpali

💻 Usage Examples

Basic Usage

import torch
from PIL import Image

from colpali_engine.models import ColQwen2, ColQwen2Processor

model = ColQwen2.from_pretrained(
        "tsystems/colqwen2-2b-v1.0",
        torch_dtype=torch.bfloat16,
        device_map="cuda:0",  # or "mps" if on Apple Silicon
    ).eval()
processor = ColQwen2Processor.from_pretrained("tsystems/colqwen2-2b-v1.0")

# Your inputs
images = [
    Image.new("RGB", (32, 32), color="white"),
    Image.new("RGB", (16, 16), color="black"),
]
queries = [
    "Is attention really all you need?",
    "What is the amount of bananas farmed in Salvador?",
]

# Process the inputs
batch_images = processor.process_images(images).to(model.device)
batch_queries = processor.process_queries(queries).to(model.device)

# Forward pass
with torch.no_grad():
    image_embeddings = model(**batch_images)
    query_embeddings = model(**batch_queries)

scores = processor.score_multi_vector(query_embeddings, image_embeddings)

🔧 Technical Details

Version specificity

This model takes dynamic image resolutions in input and does not resize them, changing their aspect ratio as in ColPali. Maximal resolution is set so that 1024 image patches are created at most. Experiments show clear improvements with larger amounts of image patches, at the cost of memory requirements. This version is trained with colpali-engine==0.3.4. Data is the same as the ColPali data described in the paper.

Model Training

Parameters

We train models use low-rank adapters (LoRA) with alpha=128 and r=128 on the transformer layers from the language model, as well as the final randomly initialized projection layer, and use a paged_adamw_8bit optimizer. We train on an 8xH100 GPU setup with distributed data parallelism (via accelerate), a learning rate of 2e-4 with linear decay with 1% warmup steps, batch size per device is 128, in bfloat16 format.

📄 License

ColQwen2's vision language backbone model (Qwen2-VL) is under apache2.0 license. This fine-tuned adapter is under CC BY NC 4.0 license. Therefore, the use of the model is research only at the moment.

📚 Documentation

Limitations

• Focus: The model primarily focuses on PDF-type documents and high-ressources languages, potentially limiting its generalization to other document types or less represented languages.
• Support: The model relies on multi-vector retreiving derived from the ColBERT late interaction mechanism, which may require engineering efforts to adapt to widely used vector retrieval frameworks that lack native multi-vector support.

Citation

If you use this models from this organization in your research, please cite the original paper as follows:

@misc{faysse2024colpaliefficientdocumentretrieval,
  title={ColPali: Efficient Document Retrieval with Vision Language Models}, 
  author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo},
  year={2024},
  eprint={2407.01449},
  archivePrefix={arXiv},
  primaryClass={cs.IR},
  url={https://arxiv.org/abs/2407.01449}, 
}

📋 Model Information