Nvcomposer

🚀 NVComposer

NVComposer is a novel approach for generative multi - view novel view synthesis (NVS) that eliminates the need for explicit external alignment, improving flexibility and accessibility.

🚀 Quick Start

NVComposer is a groundbreaking solution in the field of generative multi - view novel view synthesis (NVS). It overcomes the limitations of existing methods by removing the requirement for explicit external alignment.

✨ Features

• Eliminates External Alignment: NVComposer enables the generative model to implicitly infer spatial and geometric relationships between multiple conditional views, getting rid of the need for external multi - view alignment processes like explicit pose estimation or pre - reconstruction.
• Image - Pose Dual - Stream Diffusion Model: Simultaneously generates target novel views and condition camera poses.
• Geometry - Aware Feature Alignment Module: Distills geometric priors from dense stereo models during training.

📚 Documentation

Abstract

Recent advancements in generative models have significantly improved novel view synthesis (NVS) from multi - view data. However, existing methods depend on external multi - view alignment processes, such as explicit pose estimation or pre - reconstruction, which limits their flexibility and accessibility, especially when alignment is unstable due to insufficient overlap or occlusions between views. In this paper, we propose NVComposer, a novel approach that eliminates the need for explicit external alignment. NVComposer enables the generative model to implicitly infer spatial and geometric relationships between multiple conditional views by introducing two key components: 1) an image - pose dual - stream diffusion model that simultaneously generates target novel views and condition camera poses, and 2) a geometry - aware feature alignment module that distills geometric priors from dense stereo models during training. Extensive experiments demonstrate that NVComposer achieves state - of - the - art performance in generative multi - view NVS tasks, removing the reliance on external alignment and thus improving model accessibility. Our approach shows substantial improvements in synthesis quality as the number of unposed input views increases, highlighting its potential for more flexible and accessible generative NVS systems.

Method

NVComposer contains 1) an image - pose dual - stream diffusion model that generates novel views while implicitly estimating camera poses for conditional images, and 2) a geometry - aware feature alignment adapter that uses geometric priors distilled from pretrained dense stereo models.

📦 Installation

Download the model checkpoint using huggingface_hub (Version 0.1 as example):

from huggingface_hub import hf_hub_download

checkpoint_path = hf_hub_download(
    repo_id="TencentARC/NVComposer",
    filename="NVComposer-V0.1.ckpt"
)

The downloaded checkpoint file can be found at checkpoint_path.

💻 Usage Examples

Basic Usage

The basic usage of NVComposer involves downloading the model checkpoint as shown in the installation section.

Advanced Usage

For more advanced usage scenarios, please refer to the GitHub repository for detailed code and examples.

📄 License

The project is licensed under the other license.

Code

Please check our GitHub repository for code.

Video

Watch the introduction video in our project page.

Demo

You can try the demo here.