Vica

🚀 ViCA-7B: Visuospatial Cognitive Assistant

ViCA-7B is a vision - language model fine - tuned for visuospatial reasoning in indoor video environments, offering high - performance solutions for visual question - answering tasks.

You may also be interested in our other project, ViCA2. Please refer to the following links:

🚀 Quick Start

This README provides a comprehensive introduction to the ViCA - 7B model, including its architecture, training, evaluation, and more. For detailed usage, please refer to the relevant sections below.

✨ Features

• Multimodal Capability: Specialized for visuospatial reasoning in indoor video environments, integrating video and text information.
• State - of - the - Art Performance: Achieves excellent results on the VSI - Bench benchmark, outperforming many proprietary and open - source models.
• Interpretable Reasoning: Supports the generation of step - by - step reasoning traces, enhancing the interpretability of responses.

📚 Documentation

Overview

ViCA-7B is a vision - language model specifically fine - tuned for visuospatial reasoning in indoor video environments. Built upon the LLaVA - Video - 7B - Qwen2 architecture, it is trained using our newly proposed ViCA - 322K dataset, which emphasizes both structured spatial annotations and complex instruction - based reasoning tasks.

ViCA - 7B achieves state - of - the - art performance on [VSI - Bench](https://github.com/vision - x - nyu/thinking - in - space), outperforming both proprietary models like GPT - 4o and Gemini - 1.5 Pro, as well as larger open - source baselines.

ViCA - 7B sets a new standard for open - source multimodal spatial reasoning on indoor videos, making it a strong candidate for embodied AI and robotics use cases.

Figure 1: Performance comparison of ViCA - 7B and other models on VSI - Bench.

Model Architecture and Training Strategy

ViCA - 7B is built upon the [LLaVA - NeXT](https://github.com/LLaVA - VL/LLaVA - NeXT) framework, using Qwen2 - 7B as the language backbone and SigLIP as the visual encoder.

Key Training Features

• Fixed - Length Visual Tokenization
  Each video is uniformly sampled into 64 frames, and each frame is encoded into 210 visual tokens, resulting in a total of 13,440 visual tokens per example. This fixed - length design ensures consistent memory usage and stable optimization across batches.

• Multimodal Alignment via Lightweight Projector
  A simple MLP - based projector maps visual embeddings into the language embedding space, enabling effective fusion between video content and textual prompts during both training and inference.

• Efficient Distributed Training with DeepSpeed
  Training is conducted using DeepSpeed ZeRO - 3 Offload on 8× NVIDIA H100 80GB GPUs, with full parameter and optimizer state partitioning across devices. This setup supports large batch sizes and minimizes GPU memory overhead.

• Mixed - Precision Computation (fp16)
  We adopt mixed - precision training (fp16) to accelerate computation and reduce memory usage, without compromising accuracy. This is combined with ZeRO - 3 partitioning to further enhance training scalability.

The training was conducted over 55 hours, covering both base and complex spatial reasoning subsets.

Training Dynamics

Figure 2: Training loss, learning rate schedule, and gradient norm curves during ViCA - 7B fine - tuning. These curves illustrate a stable optimization process and smooth convergence under the DeepSpeed ZeRO - 3 setup.

Dataset

ViCA - 7B is fine - tuned on two complementary datasets:

• [ViCA - 322K](https://huggingface.co/datasets/nkkbr/ViCA - 322K):
  A large - scale dataset covering both base spatial reasoning tasks (e.g., object distance, size, count, appearance order) and complex spatial reasoning tasks involving natural language questions and scene understanding. This dataset forms the core of the model's spatial reasoning capabilities.

• [ViCA - thinking - 2.68k](https://huggingface.co/datasets/nkkbr/ViCA - thinking - 2.68k):
  A focused dataset used for instruction tuning to enhance the model's ability to generate step - by - step reasoning traces before outputting final answers. This supports more interpretable and cognitively - aligned response generation.

For details, please refer to the individual dataset pages linked above.

Evaluation: VSI - BENCH Benchmark

Figure 3: Quantitative comparison of ViCA - 7B and baseline models on VSI - Bench. ViCA - 7B achieves the best overall performance across both numerical and multiple - choice tasks.

Effect of CSR Data

CSR(Complex Spatial Reasoning) boosts generalization and accelerates learning, with notable performance jumps at intermediate checkpoints (e.g., +2.02 at 50–55%).

Data Scale vs. Performance

Performance improves significantly between 5% → 60% of data usage. After 80%, improvements plateau, indicating dataset is well - matched to model capacity.

Figure 4: Performance of ViCA - 7B under varying training data sizes (from 5% to 100%). The full dataset (including Complex Spatial Reasoning, CSR) consistently outperforms the base - only configuration. Notably, the CSR - enhanced model shows a +2.02 score jump between 50% and 55%, and a final performance gain of +4.75 at full scale. Performance plateaus beyond 80%, indicating the dataset is well - aligned with the model capacity.

Intermediate Checkpoints and Evaluation Outputs

To support detailed analysis and reproducibility, we provide two sets of intermediate checkpoints saved at every 5% increment of the training data. These models are trained for a single epoch and are useful for understanding how performance evolves as training progresses.

We also release the corresponding raw evaluation outputs (e.g., .json prediction files) for each checkpoint. The evaluation script used to produce these outputs is available in our GitHub repository.

Full Dataset (ViCA - 322K: Base + CSR)

This series corresponds to the full training set, including both base spatial reasoning and complex spatial reasoning (CSR):

Raw evaluation outputs are available [here](https://huggingface.co/nkkbr/ViCA/tree/main/raw_evaluation_outputs/vsi - bench_all_data/).

Base - only Subset (ViCA - 322K: Base)

This series is trained only on the base spatial reasoning subset of ViCA - 322K, without any CSR examples:

Raw evaluation outputs are available [here](https://huggingface.co/nkkbr/ViCA/tree/main/raw_evaluation_outputs/vsi - bench_only_base/).

Source - wise Checkpoints

While the full ViCA - 322K dataset was curated by us, the underlying videos and associated metadata are sourced from three distinct indoor video datasets:

• ARKitScenes
• [ScanNet](http://www.scan - net.org)
• ScanNet++

To better understand how each source contributes to model performance, we fine - tuned ViCA - 7B on subsets of ViCA - 322K that exclusively use data from each source. For each subset, we provide checkpoints trained with 10% increments of the available data, from 10% to 100%.

Corresponding raw evaluation outputs (e.g., .json predictions) are also provided for all checkpoints.

ARKitScenes - Only Checkpoints

[The content about ARKitScenes - Only Checkpoints in the original text seems incomplete. Please provide the full content if you need a more complete translation.]

🔧 Technical Details

Model Index

Results on VSI - Bench

📄 License

This project is licensed under the Apache - 2.0 license.