Slimsam 77 Uniform

🚀 SlimSAM-77 with Transformers.js

This project provides the ONNX weights of the nielsr/slimsam-77-uniform base model, making it compatible with the transformers.js library. It enables seamless mask generation and subject isolation in a web environment.

🚀 Quick Start

Prerequisites

If you haven't already, you can install the Transformers.js JavaScript library from NPM using:

npm i @huggingface/transformers

✨ Features

• Compatibility with transformers.js for web-based usage.
• Mask generation and subject isolation capabilities.
• Online demo available for quick testing.

📦 Installation

npm i @huggingface/transformers

💻 Usage Examples

Basic Usage

import { SamModel, AutoProcessor, RawImage } from '@huggingface/transformers';

// Load model and processor
const model = await SamModel.from_pretrained('Xenova/slimsam-77-uniform');
const processor = await AutoProcessor.from_pretrained('Xenova/slimsam-77-uniform');

// Prepare image and input points
const img_url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/corgi.jpg';
const raw_image = await RawImage.read(img_url);
const input_points = [[[340, 250]]];

// Process inputs and perform mask generation
const inputs = await processor(raw_image, { input_points });
const outputs = await model(inputs);

// Post-process masks
const masks = await processor.post_process_masks(outputs.pred_masks, inputs.original_sizes, inputs.reshaped_input_sizes);
console.log(masks);
// [
//   Tensor {
//     dims: [ 1, 3, 410, 614 ],
//     type: 'bool',
//     data: Uint8Array(755220) [ ... ],
//     size: 755220
//   }
// ]
const scores = outputs.iou_scores;
console.log(scores);
// Tensor {
//   dims: [ 1, 1, 3 ],
//   type: 'float32',
//   data: Float32Array(3) [
//     0.8350210189819336,
//     0.9786665439605713,
//     0.8379436731338501
//   ],
//   size: 3
// }

Advanced Usage

You can then visualize the generated mask with:

const image = RawImage.fromTensor(masks[0][0].mul(255));
image.save('mask.png');

Next, select the channel with the highest IoU score, which in this case is the second (green) channel. Intersecting this with the original image gives us an isolated version of the subject:

📚 Documentation

Demo

We've also got an online demo, which you can try out here.

⚠️ Important Note

Having a separate repo for ONNX weights is intended to be a temporary solution until WebML gains more traction. If you would like to make your models web-ready, we recommend converting to ONNX using 🤗 Optimum and structuring your repo like this one (with ONNX weights located in a subfolder named onnx).