PVD 160k Mistral 7b

🚀 Text-Based Reasoning About Vector Graphics

This project aims to address the challenges of large multimodal models in reasoning tasks related to vector graphics by proposing a visual reasoning framework, VDLM, which uses text-based visual descriptions and outperforms state - of - the - art models.

🚀 Quick Start

We observe that current large multimodal models (LMMs) still struggle with seemingly straightforward reasoning tasks that require precise perception of low - level visual details, such as identifying spatial relations or solving simple mazes. In particular, this failure mode persists in question - answering tasks about vector graphics—images composed purely of 2D objects and shapes.

To solve this challenge, we propose Visually Descriptive Language Model (VDLM), a visual reasoning framework that operates with intermediate text - based visual descriptions—SVG representations and learned Primal Visual Description, which can be directly integrated into existing LLMs and LMMs. We demonstrate that VDLM outperforms state - of - the - art large multimodal models, such as GPT - 4V, across various multimodal reasoning tasks involving vector graphics. See our paper for more details.

✨ Features

• Addressing LMM Limitations: VDLM tackles the challenges faced by large multimodal models in reasoning tasks related to vector graphics.
• Text - Based Visual Descriptions: It uses SVG representations and learned Primal Visual Description for visual reasoning.
• Performance Advantage: Outperforms state - of - the - art large multimodal models like GPT - 4V in multimodal reasoning tasks involving vector graphics.

📄 License

This project is licensed under the Apache - 2.0 license.

📚 Documentation

Useful Links

• 🌐 Homepage
• 📃 Paper
• 🤗 Data (PVD - 160k)
• 🤗 Model (PVD - 160k - Mistral - 7b)
• 💻 Code

Information Table