CoreML - DETR - Semantic Segmentation Open - source Model: A Powerful Tool for Efficient Semantic Segmentation Deployment on Mobile Devices

Coreml Detr Semantic Segmentation

Developed by apple

DETR-Resnet50 is a Transformer-based semantic segmentation model, providing efficient mobile deployment capabilities through the Core ML format.

Image Segmentation Open Source License:Apache-2.0 #Transformer semantic segmentation #End-to-end object detection #Neural engine optimization

Downloads 91

Release Time : 5/16/2024

Model Overview

This model is a Core ML implementation of DEtection TRansformer (DETR), specifically designed for semantic segmentation tasks. It combines the advantages of convolutional neural networks and Transformer architecture to efficiently perform image segmentation.

Model Features

End-to-end Transformer architecture

Uses Transformer architecture for object detection and segmentation, avoiding complex post-processing steps in traditional methods.

Efficient mobile deployment

Provides Core ML format models, supporting efficient operation on iOS/macOS devices.

Multi-precision support

Offers both Float32 and Float16 precision versions to balance accuracy and performance requirements.

Model Capabilities

Image semantic segmentation

Object detection

Mobile image processing

Use Cases

Computer vision

Image scene understanding

Performs semantic segmentation labeling of different objects and regions in an image

Achieves 0.393 IoU and 0.746 pixel accuracy on the COCO dataset

Mobile image analysis

Executes real-time image segmentation tasks on mobile devices like iPhone

Achieves 40ms inference speed on iPhone 15 Pro Max

🚀 DETR-Resnet50 (semantic segmentation) Core ML Models

This project provides DETR-Resnet50 Core ML models for semantic segmentation, trained on COCO 2017 object detection dataset.

🚀 Quick Start

See the Files tab for converted models.

DEtection TRansformer (DETR) model trained end-to-end on COCO 2017 object detection (118k annotated images). It was introduced in the paper End-to-End Object Detection with Transformers by Carion et al. and first released in this repository.

✨ Features

The DETR model is an encoder-decoder transformer with a convolutional backbone. Two heads are added on top of the decoder outputs in order to perform object detection: a linear layer for the class labels and a MLP (multi-layer perceptron) for the bounding boxes. The model uses so-called object queries to detect objects in an image. Each object query looks for a particular object in the image. For COCO, the number of object queries is set to 100.

The model is trained using a "bipartite matching loss": one compares the predicted classes + bounding boxes of each of the N = 100 object queries to the ground truth annotations, padded up to the same length N (so if an image only contains 4 objects, 96 annotations will just have a "no object" as class and "no bounding box" as bounding box). The Hungarian matching algorithm is used to create an optimal one-to-one mapping between each of the N queries and each of the N annotations. Next, standard cross-entropy (for the classes) and a linear combination of the L1 and generalized IoU loss (for the bounding boxes) are used to optimize the parameters of the model.

model image

📚 Documentation

Evaluation - Variants

Variant	Parameters	Size (MB)	Weight Precision	Act. Precision	IoU	Pixel accuracy
facebook/detr-resnet-50-panoptic (PyTorch)	43M	172	Float32	Float32	0.393	0.746
DETRResnet50SemanticSegmentationF32	43M	171	Float32	Float32	0.393	0.746
DETRResnet50SemanticSegmentationF16	43M	86	Float16	Float16	0.395	0.746

IoU and Pixel accuracy measured on 512 images from the COCO dataset. The ground truth labels were extracted from the panoptic segmentation annotations, transformed to semantic segmentation masks. Input images were resized so that the smaller edge equals 448, then center-cropped.

Inference time

The following results refer to DETRResnet50SemanticSegmentationF16. The compute units for MacBook Pro (M1 Max) were manually selected to "CPU and Neural Engine".

Device	OS	Inference time (ms)	Dominant compute unit
iPhone 15 Pro Max	17.5	40	Neural Engine
MacBook Pro (M1 Max)	14.5	43	Neural Engine
iPhone 12 Pro Max	18.0	52	Neural Engine
MacBook Pro (M3 Max)	15.0	29	Neural Engine

📦 Installation

Download

Install huggingface-cli

brew install huggingface-cli

To download one of the .mlpackage folders to the models directory:

huggingface-cli download \
  --local-dir models --local-dir-use-symlinks False \
  apple/coreml-detr-semantic-segmentation \
  --include "DETRResnet50SemanticSegmentationF16.mlpackage/*"

To download everything, skip the --include argument. This will retrieve float32 and float16 variants, as well as quantized versions of the float16 variant.

Integrate in Swift apps

The huggingface/coreml-examples repository contains sample Swift code for coreml-detr-semantic-segmentation and other models. See the instructions there to build the demo app, which shows how to use the model in your own Swift apps.

📄 License

This project is licensed under the apache-2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご