xclip-base-patch16-zero-shot Open Source Model - Achieving Universal Video and Language Matching and Understanding

Xclip Base Patch16 Zero Shot

Developed by aurelio-ai

X-CLIP is a minimalist extension of CLIP for general video-language understanding, trained via contrastive learning to match videos and texts.

Text-to-Video

Transformers

EnglishOpen Source License:MIT #Video-Text Contrastive Learning #Zero-Shot Video Classification #Multimodal Understanding

Downloads 22

Release Time : 11/8/2023

Model Overview

The X-CLIP model (base size, 16x16 patch resolution) is trained on Kinetics-400 and suitable for zero-shot, few-shot, or fully supervised video classification, as well as video-text retrieval tasks.

Model Features

Zero-Shot Video Classification

Directly applicable to video classification tasks without fine-tuning, supporting zero-shot learning.

Video-Text Matching

Capable of assessing the match between text descriptions and given video content.

Multi-Task Support

Supports various tasks including video classification and video-text retrieval.

Model Capabilities

Video Classification

Video-Text Retrieval

Zero-Shot Learning

Use Cases

Video Understanding

Video Classification

Classify video content, such as action recognition and scene recognition.

Zero-shot top-1 accuracy: 44.6% on HMDB-51, 72.0% on UCF-101, and 65.2% on Kinetics-600.

Video-Text Retrieval

Retrieve relevant video content based on text descriptions.

🚀 X-CLIP (base-sized model)

X-CLIP is a model for general video - language understanding. It extends CLIP and is trained on video - text pairs, enabling tasks like zero - shot, few - shot or fully supervised video classification and video - text retrieval.

🚀 Quick Start

The X - CLIP model (base - sized, patch resolution of 16) is trained on Kinetics - 400. It was introduced in the paper Expanding Language - Image Pretrained Models for General Video Recognition by Ni et al. and first released in [this repository](https://github.com/microsoft/VideoX/tree/master/X - CLIP).

This model was trained using 32 frames per video, at a resolution of 224x224.

Disclaimer: The team releasing X - CLIP did not write a model card for this model so this model card has been written by the Hugging Face team.

✨ Features

X - CLIP is a minimal extension of CLIP for general video - language understanding.
The model is trained in a contrastive way on (video, text) pairs, allowing it to be used for tasks like zero - shot, few - shot or fully supervised video classification and video - text retrieval.

📚 Documentation

Intended uses & limitations

You can use the raw model for determining how well text goes with a given video. See the model hub to look for fine - tuned versions on a task that interests you.

How to use

For code examples, we refer to the documentation.

Training data

This model was trained on Kinetics 400.

Preprocessing

The exact details of preprocessing during training can be found [here](https://github.com/microsoft/VideoX/blob/40f6d177e0a057a50ac69ac1de6b5938fd268601/X - CLIP/datasets/build.py#L247).

The exact details of preprocessing during validation can be found [here](https://github.com/microsoft/VideoX/blob/40f6d177e0a057a50ac69ac1de6b5938fd268601/X - CLIP/datasets/build.py#L285).

During validation, one resizes the shorter edge of each frame, after which center cropping is performed to a fixed - size resolution (like 224x224). Next, frames are normalized across the RGB channels with the ImageNet mean and standard deviation.

Evaluation results

Property	Details
HMDB - 51 Zero - shot Top - 1 Accuracy	44.6%
UCF - 101 Zero - shot Top - 1 Accuracy	72.0%
Kinetics - 600 Zero - shot Top - 1 Accuracy	65.2%

📄 License

This model is released under the MIT license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご