Colnomic Embed Multimodal 3b

🚀 ColNomic Embed Multimodal 3B: State-of-the-Art Visual Document Retrieval

colnomic-embed-multimodal-3b is a multi-vector state-of-the-art multimodal embedding model that shines in visual document retrieval tasks. It can solve the problem of efficiently retrieving visual documents and provides high - performance and unified encoding solutions.

Model Information

✨ Features

• High Performance: Achieves 61.2 NDCG@5 on Vidore - v2, outperforming all other models except ColNomic Embed Multimodal 7B.
• Unified Text - Image Encoding: Directly encodes interleaved text and images without complex preprocessing.
• Advanced Architecture: A 3B parameter multimodal embedding model.
• Open - Weights: Model weights are available for research use.

📦 Installation

To use colnomic-embed-multimodal-3b, please install colpali from source:

pip install git+https://github.com/illuin-tech/colpali.git

💻 Usage Examples

Basic Usage

import torch
from PIL import Image
from transformers.utils.import_utils import is_flash_attn_2_available

from colpali_engine.models import ColQwen2_5, ColQwen2_5_Processor

model_name = "nomic-ai/colnomic-embed-multimodal-3b"

model = ColQwen2_5.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="cuda:0",  # or "mps" if on Apple Silicon
    attn_implementation="flash_attention_2" if is_flash_attn_2_available() else None,
).eval()

processor = ColQwen2_5_Processor.from_pretrained(model_name)

# Your inputs
images = [
    Image.new("RGB", (128, 128), color="white"),
    Image.new("RGB", (64, 32), color="black"),
]
queries = [
    "What is the organizational structure for our R&D department?",
    "Can you provide a breakdown of last year’s financial performance?",
]

# Process the inputs
batch_images = processor.process_images(images).to(model.device)
batch_queries = processor.process_queries(queries).to(model.device)

# Forward pass
with torch.no_grad():
    image_embeddings = model(**batch_images)
    query_embeddings = model(**batch_queries)

scores = processor.score_multi_vector(query_embeddings, image_embeddings)

📚 Documentation

Performance

Model Architecture

• Total Parameters: 3B
• Training Approach: Fine - tuned from Qwen2.5 - VL 3B Instruct
• Architecture Type: Vision - Language Model with unified text and image input processing
• Key Innovations:
  • Same - source sampling to create harder in - batch negatives
  • Multi - vector output option for enhanced performance

Integration with RAG Workflows

Nomic Embed Multimodal 3B seamlessly integrates with Retrieval Augmented Generation (RAG) workflows:

1. Direct Document Embedding: Skip OCR and complex processing by directly embedding document page images.
2. Faster Processing: Eliminate preprocessing steps for quicker indexing.
3. More Complete Information: Capture both textual and visual cues in a single embedding.
4. Simple Implementation: Use the same API for both text and images.

Recommended Use Cases

The model excels at handling real - world document retrieval scenarios that challenge traditional text - only systems:

• Research Papers: Capture equations, diagrams, and tables.
• Technical Documentation: Encode code blocks, flowcharts, and screenshots.
• Product Catalogs: Represent images, specifications, and pricing tables.
• Financial Reports: Embed charts, graphs, and numerical data.
• Visually Rich Content: Where layout and visual information are important.
• Multilingual Documents: Where visual context provides important cues.

Training Details

ColNomic Embed Multimodal 3B was developed through several key innovations:

1. Sampling From the Same Source: Forcing sampling from the same dataset source creates harder in - batch negatives, preventing the model from learning dataset artifacts.
2. Multi - Vector Configuration: Providing a multi - vector variant that achieves higher performance than the dense variant.

Limitations

• Performance may vary when processing documents with unconventional layouts or unusual visual elements.
• While it handles multiple languages, performance is strongest on English content.
• Processing very large or complex documents may require dividing them into smaller chunks.
• Performance on documents with handwriting or heavily stylized fonts may be reduced.

🔧 Technical Details

The model's development involves two key innovative techniques. First, the same - source sampling method creates more challenging in - batch negatives, which helps the model avoid learning dataset - specific artifacts. Second, the multi - vector configuration offers a variant that outperforms the dense variant, enhancing the overall performance of the model.

📄 License

No license information is provided in the original README.

🔗 Join the Nomic Community

• Nomic Embed Ecosystem: https://www.nomic.ai/embed
• Website: https://nomic.ai
• Twitter: https://twitter.com/nomic_ai
• Discord: https://discord.gg/myY5YDR8z8

📖 Citation

If you find this model useful in your research or applications, please consider citing:

@misc{faysse2024colpaliefficientdocumentretrieval,
  title={ColPali: Efficient Document Retrieval with Vision Language Models}, 
  author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo},
  year={2024},
  eprint={2407.01449},
  archivePrefix={arXiv},
  primaryClass={cs.IR},
  url={https://arxiv.org/abs/2407.01449}, 
}
@misc{ma2024unifyingmultimodalretrievaldocument,
      title={Unifying Multimodal Retrieval via Document Screenshot Embedding}, 
      author={Xueguang Ma and Sheng-Chieh Lin and Minghan Li and Wenhu Chen and Jimmy Lin},
      year={2024},
      eprint={2406.11251},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2406.11251}, 
}
@misc{nomicembedmultimodal2025,
  title={Nomic Embed Multimodal: Interleaved Text, Image, and Screenshots for Visual Document Retrieval},
  author={Nomic Team},
  year={2025},
  publisher={Nomic AI},
  url={https://nomic.ai/blog/posts/nomic-embed-multimodal},
}