đ Mask2Former
The Mask2Former model is trained on COCO instance segmentation (large - sized version, Swin backbone). It offers a unified approach to multiple segmentation tasks.
đ Quick Start
The Mask2Former model, trained on COCO instance segmentation, is a powerful tool for image segmentation tasks. It was introduced in a research paper and first released in a specific GitHub repository.
⨠Features
- Unified Paradigm: Addresses instance, semantic, and panoptic segmentation with the same approach by predicting masks and labels.
- Performance Improvement: Outperforms the previous SOTA, MaskFormer, in both performance and efficiency.
- Advanced Architecture: Replaces the pixel decoder with a multi - scale deformable attention Transformer and uses a masked - attention Transformer decoder.
- Training Efficiency: Calculates loss on subsampled points instead of whole masks.
đ Documentation
Model description
Mask2Former addresses instance, semantic and panoptic segmentation with the same paradigm: by predicting a set of masks and corresponding labels. Hence, all 3 tasks are treated as if they were instance segmentation. Mask2Former outperforms the previous SOTA,
MaskFormer both in terms of performance an efficiency by (i) replacing the pixel decoder with a more advanced multi - scale deformable attention Transformer, (ii) adopting a Transformer decoder with masked attention to boost performance without
without introducing additional computation and (iii) improving training efficiency by calculating the loss on subsampled points instead of whole masks.

Intended uses & limitations
You can use this particular checkpoint for instance segmentation. See the model hub to look for other
fine - tuned versions on a task that interests you.
How to use
Here is how to use this model:
import requests
import torch
from PIL import Image
from transformers import AutoImageProcessor, Mask2FormerForUniversalSegmentation
processor = AutoImageProcessor.from_pretrained("facebook/mask2former-swin-large-coco-instance")
model = Mask2FormerForUniversalSegmentation.from_pretrained("facebook/mask2former-swin-large-coco-instance")
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
class_queries_logits = outputs.class_queries_logits
masks_queries_logits = outputs.masks_queries_logits
result = processor.post_process_instance_segmentation(outputs, target_sizes=[image.size[::-1]])[0]
predicted_instance_map = result["segmentation"]
For more code examples, we refer to the documentation.
đ License
License: other
Property |
Details |
Model Type |
Mask2Former model trained on COCO instance segmentation (large - sized version, Swin backbone) |
Training Data |
COCO |
Disclaimer: The team releasing Mask2Former did not write a model card for this model so this model card has been written by the Hugging Face team.