Clip4clip Webvid150k

🚀 Model Card for CLIP4Clip/WebVid-150k

A CLIP4Clip video - text retrieval model trained on a subset of the WebVid dataset, enabling effective video - text retrieval.

🚀 Quick Start

Extracting Text Embeddings:

import numpy as np
import torch
from transformers import CLIPTokenizer, CLIPTextModelWithProjection


search_sentence = "a basketball player performing a slam dunk"

model = CLIPTextModelWithProjection.from_pretrained("Searchium-ai/clip4clip-webvid150k")
tokenizer = CLIPTokenizer.from_pretrained("Searchium-ai/clip4clip-webvid150k")

inputs = tokenizer(text=search_sentence , return_tensors="pt")
outputs = model(input_ids=inputs["input_ids"], attention_mask=inputs["attention_mask"])

# Normalize embeddings for retrieval:
final_output = outputs[0] / outputs[0].norm(dim=-1, keepdim=True)
final_output = final_output.cpu().detach().numpy()
print("final output: ", final_output)

Extracting Video Embeddings:

An additional notebook "GSI_VideoRetrieval_VideoEmbedding.ipynb", provides instructions for extracting video embeddings and includes the necessary tools for preprocessing videos.

✨ Features

• Video - Text Retrieval: A CLIP4Clip video - text retrieval model trained on a subset of the WebVid dataset.
• Based on CLIP Architecture: Built on the [clip - vit - base - patch32](https://huggingface.co/openai/clip - vit - base - patch32) architecture.
• Large - Scale Training: Trained on a subset of 150,000 video - text pairs from the WebVid dataset.

📚 Documentation

Model Details

A CLIP4Clip video - text retrieval model trained on a subset of the WebVid dataset. The model and training method are described in the paper "Clip4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval" by Lou et el, and implemented in the accompanying GitHub repository.

The training process utilized the [WebVid Dataset](https://m - bain.github.io/webvid - dataset/), a comprehensive collection of short videos with corresponding textual descriptions sourced from the web. For training purposes, a subset consisting of the first 150,000 video - text pairs from the dataset were used.

This HF model is based on the [clip - vit - base - patch32](https://huggingface.co/openai/clip - vit - base - patch32) architecture, with weights trained by Daphna Idelson at Searchium.

Model Intended Use

This model is intended for use in large - scale video - text retrieval applications.

To illustrate its functionality, refer to the accompanying [Video Search Space](https://huggingface.co/spaces/Searchium - ai/Video - Search) which provides a search demonstration on a vast collection of approximately 1.5 million videos. This interactive demo showcases the model's capability to effectively retrieve videos based on text queries, highlighting its potential for handling substantial video datasets.

Motivation

As per the original authors, the main motivation behind this work is to leverage the power of the CLIP image - language pre - training model and apply it to learning visual - temporal concepts from videos, thereby improving video - based searches.

By using the WebVid dataset, the model's capabilities were enhanced even beyond those described in the paper, thanks to the large - scale and diverse nature of the dataset empowering the model's performance.

Evaluations

To evaluate the model's performance we used the last 10,000 video clips and their accompanying text from the Webvid dataset. We evaluate R1, R5, R10, MedianR, and MeanR on:

1. Zero - shot pretrained clip - vit - base - patch32 model
2. CLIP4Clip based weights trained on the dataset [MSR - VTT](https://paperswithcode.com/dataset/msr - vtt), consisting of 10,000 video - text pairs
3. CLIP4Clip based weights trained on a 150K subset of the dataset Webvid - 2M
4. CLIP4Clip based weights trained on a 150K subset of the dataset Webvid - 2M - binarized and further finetuned on 100 top searches - for search acceleration and efficiency [1].

For an elaborate description of the evaluation refer to the notebook [GSI_VideoRetrieval - Evaluation](https://huggingface.co/Searchium - ai/clip4clip - webvid150k/blob/main/Notebooks/GSI_VideoRetrieval - Evaluation.ipynb).

Acknowledgements

Acknowledging Diana Mazenko of Searchium for adapting and loading the model to Hugging Face, and for creating a Hugging Face [SPACE](https://huggingface.co/spaces/Searchium - ai/Video - Search) for a large - scale video - search demo.

Acknowledgments also to Lou et el for their comprehensive work on CLIP4Clip and openly available code.

Citations

CLIP4Clip paper

@Article{Luo2021CLIP4Clip,
  author  = {Huaishao Luo and Lei Ji and Ming Zhong and Yang Chen and Wen Lei and Nan Duan and Tianrui Li},
  title   = {{CLIP4Clip}: An Empirical Study of CLIP for End to End Video Clip Retrieval},
  journal = {arXiv preprint arXiv:2104.08860},
  year    = {2021},
}

OpenAI CLIP paper

@inproceedings{Radford2021LearningTV,
  title={Learning Transferable Visual Models From Natural Language Supervision},
  author={Alec Radford and Jong Wook Kim and Chris Hallacy and A. Ramesh and Gabriel Goh and Sandhini Agarwal and Girish Sastry and Amanda Askell and Pamela Mishkin and Jack Clark and Gretchen Krueger and Ilya Sutskever},
  booktitle={ICML},
  year={2021}
}