Deeplabv3 Mobilevit Xx Small

🚀 MobileViT + DeepLabV3 (Extra Extra Small-sized Model)

A MobileViT model pre-trained on PASCAL VOC at 512x512 resolution for semantic segmentation.

🚀 Quick Start

The MobileViT model was pre-trained on PASCAL VOC at a resolution of 512x512. It was introduced in MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer by Sachin Mehta and Mohammad Rastegari, and first released in this repository. The license used is Apple sample code license.

Disclaimer: The team releasing MobileViT did not write a model card for this model, so this model card has been written by the Hugging Face team.

✨ Features

• MobileViT is a light-weight, low latency convolutional neural network. It combines MobileNetV2-style layers with a new block that replaces local processing in convolutions with global processing using transformers.
• The image data is converted into flattened patches before being processed by the transformer layers and then "unflattened" back into feature maps.
• MobileViT does not require any positional embeddings.
• The model in this repo adds a DeepLabV3 head to the MobileViT backbone for semantic segmentation.

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

from transformers import MobileViTFeatureExtractor, MobileViTForSemanticSegmentation
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

feature_extractor = MobileViTFeatureExtractor.from_pretrained("apple/deeplabv3-mobilevit-xx-small")
model = MobileViTForSemanticSegmentation.from_pretrained("apple/deeplabv3-mobilevit-xx-small")

inputs = feature_extractor(images=image, return_tensors="pt")

outputs = model(**inputs)
logits = outputs.logits
predicted_mask = logits.argmax(1).squeeze(0)

Currently, both the feature extractor and model support PyTorch.

📚 Documentation

Intended uses & limitations

You can use the raw model for semantic segmentation. See the model hub to look for fine-tuned versions on a task that interests you.

Model description

MobileViT is a light-weight, low latency convolutional neural network that combines MobileNetV2-style layers with a new block that replaces local processing in convolutions with global processing using transformers. As with ViT (Vision Transformer), the image data is converted into flattened patches before it is processed by the transformer layers. Afterwards, the patches are "unflattened" back into feature maps. This allows the MobileViT-block to be placed anywhere inside a CNN. MobileViT does not require any positional embeddings.

The model in this repo adds a DeepLabV3 head to the MobileViT backbone for semantic segmentation.

Training data

The MobileViT + DeepLabV3 model was pretrained on ImageNet-1k, a dataset consisting of 1 million images and 1,000 classes, and then fine-tuned on the PASCAL VOC2012 dataset.

Training procedure

Preprocessing

At inference time, images are center-cropped at 512x512. Pixels are normalized to the range [0, 1]. Images are expected to be in BGR pixel order, not RGB.

Pretraining

The MobileViT networks are trained from scratch for 300 epochs on ImageNet-1k on 8 NVIDIA GPUs with an effective batch size of 1024 and learning rate warmup for 3k steps, followed by cosine annealing. Also used were label smoothing cross-entropy loss and L2 weight decay. Training resolution varies from 160x160 to 320x320, using multi-scale sampling.

To obtain the DeepLabV3 model, MobileViT was fine-tuned on the PASCAL VOC dataset using 4 NVIDIA A100 GPUs.

Evaluation results

BibTeX entry and citation info

@inproceedings{vision-transformer,
title = {MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer},
author = {Sachin Mehta and Mohammad Rastegari},
year = {2022},
URL = {https://arxiv.org/abs/2110.02178}
}

🔧 Technical Details

The technical details are scattered throughout the "Model description", "Training procedure" sections, etc. The key points include:

• MobileViT combines MobileNetV2-style layers with a new block using transformers for global processing.
• Image data is converted to flattened patches and then back to feature maps.
• Training involves pre - training on ImageNet - 1k and fine - tuning on PASCAL VOC.
• Specific training parameters such as epochs, batch size, learning rate schedule, and loss functions are used.

📄 License

The license used is Apple sample code license.