Open-source sam2-hiera-tiny Model - An Efficient Image and Video Prompt-based Visual Segmentation Tool

Sam2 Hiera Tiny

Developed by facebook

SAM 2 is a foundational model for promptable visual segmentation in images and videos developed by FAIR, supporting efficient segmentation through prompts.

Image Segmentation Open Source License:Apache-2.0 #Promptable Segmentation #Video Object Tracking #Zero-shot Learning

Downloads 41.88k

Release Time : 8/2/2024

Model Overview

SAM 2 is an advanced visual segmentation model capable of quickly generating high-quality segmentation masks in images and videos based on user-provided prompts (such as points or boxes).

Model Features

Multimodal Prompt Support

Supports interactive segmentation through various prompt methods such as points and boxes.

Image and Video Versatility

The same model architecture can handle both image and video segmentation tasks.

Efficient Inference

Supports bfloat16 precision and CUDA acceleration for fast inference.

Real-time Propagation

Enables real-time prompt propagation and object tracking during video processing.

Model Capabilities

Image Segmentation

Video Object Segmentation

Interactive Segmentation

Mask Generation

Use Cases

Computer Vision

Image Editing

Quickly isolate objects in images for editing.

High-quality object segmentation masks.

Video Analysis

Track specific objects in videos.

Consistent object segmentation across frames.

Augmented Reality

AR Content Overlay

Real-time segmentation of objects in real-world scenes.

Provides precise object boundaries for AR applications.

🚀 SAM 2: Segment Anything in Images and Videos

This repository hosts SAM 2, a foundation model developed by FAIR for promptable visual segmentation in both images and videos. For more detailed information, please refer to the SAM 2 paper.

The official code has been publicly released in this repo.

🚀 Quick Start

This section provides a quick overview of how to use SAM 2 for image and video prediction.

💻 Usage Examples

Basic Usage

Image Prediction

import torch
from sam2.sam2_image_predictor import SAM2ImagePredictor

predictor = SAM2ImagePredictor.from_pretrained("facebook/sam2-hiera-tiny")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    predictor.set_image(<your_image>)
    masks, _, _ = predictor.predict(<input_prompts>)

Video Prediction

import torch
from sam2.sam2_video_predictor import SAM2VideoPredictor

predictor = SAM2VideoPredictor.from_pretrained("facebook/sam2-hiera-tiny")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state(<your_video>)

    # add new prompts and instantly get the output on the same frame
    frame_idx, object_ids, masks = predictor.add_new_points_or_box(state, <your_prompts>):

    # propagate the prompts to get masklets throughout the video
    for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
        ...

For more detailed usage, please refer to the demo notebooks.

Citation

To cite the paper, model, or software, please use the following BibTeX entry:

@article{ravi2024sam2,
  title={SAM 2: Segment Anything in Images and Videos},
  author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph},
  journal={arXiv preprint arXiv:2408.00714},
  url={https://arxiv.org/abs/2408.00714},
  year={2024}
}

📄 License

This project is licensed under the Apache-2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご