In2in

🚀 in2IN: Leveraging individual Information to Generate Human INteractions

A novel diffusion model for generating human - human motion interactions, which not only conditions on the overall textual description but also individual descriptions of actions.

🚀 Quick Start

Generating human - human motion interactions based on textual descriptions is highly useful in areas like robotics, gaming, animation, and the metaverse. However, it's challenging to model high - dimensional inter - personal dynamics and capture intra - personal diversity. Current methods have limitations in generating diverse intra - personal dynamics due to dataset and conditioning strategy constraints.

We introduce in2IN, a diffusion model for human - human motion generation. It's conditioned on both the overall interaction's textual description and individual descriptions of each person's actions. We use a large language model to extend the InterHuman dataset with individual descriptions for training. in2IN achieves state - of - the - art performance on the InterHuman dataset.

To increase intra - personal diversity in existing interaction datasets, we propose DualMDM, a model composition technique. It combines motions generated by in2IN and a single - person motion prior pre - trained on HumanML3D. DualMDM generates motions with higher individual diversity, better controls intra - personal dynamics, and maintains inter - personal coherence.

💻 Usage Examples

Basic Usage

from transformers import AutoModel        
model = AutoModel.from_pretrained("pabloruizponce/in2IN", trust_remote_code=True)
model(textI, texti1, texti2)

Details

Input: The model takes as input the textual description of the overall interaction and the two individual descriptions from the interactants. Output (2,T,N,3): The model returns an array with the coordinates of the N joints of each interactant during a motion of T timesteps of duration.

📚 Documentation

Citation

If you find our work helpful, please cite:

@InProceedings{Ruiz-Ponce_2024_CVPR,
    author    = {Ruiz-Ponce, Pablo and Barquero, German and Palmero, Cristina and Escalera, Sergio and Garc{\'\i}a-Rodr{\'\i}guez, Jos\'e},
    title     = {in2IN: Leveraging Individual Information to Generate Human INteractions},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2024},
    pages     = {1941-1951}
}