Paligemma-3b-ft-widgetcap-waveui-448 Open-source Vision-Language Model - Free Deployment for Efficient Object Detection

Paligemma 3b Ft Widgetcap Waveui 448

Developed by agentsea

A vision-language model fine-tuned for object detection tasks on the WaveUI dataset, based on PaliGemma 3B 448-resolution weights

Image-to-Text

Transformers

English#UI Element Detection #High-Precision Bounding Box #Agent Construction Tool

Downloads 344

Release Time : 7/8/2024

Model Overview

A vision-language model focused on UI element detection, serving as a key component of the AgentSea open-source agent construction toolkit

Model Features

High-Precision UI Element Detection

Fine-tuned on the WaveUI dataset, specifically optimized for UI element detection performance

PaliGemma Architecture-Based

Built upon Google's PaliGemma 3B model with powerful multimodal understanding capabilities

Open-Source Agent Support

Serves as a core component of the AgentSea open-source agent construction toolkit

Model Capabilities

UI Element Detection

Multimodal Understanding

Object Localization

Use Cases

UI Automation

Interface Element Recognition

Automatically identifies interface elements like buttons and input fields in applications

Achieves 0.40 IoU on test set

Agent Development

Automated Testing

Used to build testing agents capable of understanding UIs

🚀 Paligemma WaveUI

This project leverages the Transformers library to fine - tune the PaliGemma 3B 448 - res weights on the WaveUI dataset for object - detection tasks, aiming to enhance UI element detection capabilities.

🚀 Quick Start

To start using this model, run the following:

from transformers import AutoProcessor, PaliGemmaForConditionalGeneration

model = PaliGemmaForConditionalGeneration.from_pretrained("agentsea/paligemma-3b-ft-widgetcap-waveui-448").eval()
processor = AutoProcessor.from_pretrained("agentsea/paligemma-3b-ft-widgetcap-waveui-448")

✨ Features

Fine - tuned on the WaveUI dataset for object detection, especially focusing on UI element detection.
Part of the open - source toolkit for building agents at AgentSea.

📚 Documentation

Model Details

Model Description

This fine - tune was done atop of the Paligemma 448 Widgetcap model, using the WaveUI dataset, which contains ~80k examples of labeled UI elements.

The fine - tune was done for the object detection task. Specifically, this model aims to perform well at UI element detection, as part of a wider effort to enable our open - source toolkit for building agents at AgentSea.

Property	Details
Developed by	https://agentsea.ai/
Language(s) (NLP)	en
Finetuned from model	https://huggingface.co/google/paligemma-3b-ft-widgetcap-448

Demo

You can find a demo for this model here.

Notes

⚠️ Important Note

The only task used in the fine - tune was the object detection task, so it might not perform well in other types of tasks.

Data

We used the WaveUI dataset for this fine - tune. Before using it, we preprocessed the data to use the Paligemma bounding - box format.

Evaluation

We calculated the mean IoU over 1024 examples of the test set using 3 different closed - source models: Gemini 1.5 Pro, Claude 3.5 Sonnet and GPT 4o. We also ran this same calculation using the PaliGemma WaveUI fine - tunes. We obtained the following values:

Gemini 1.5 Pro: 0.12
Claude 3.5 Sonnet: 0.05
GPT 4o: 0.05
PaliGemma Widgetcap+WaveUI 448: 0.40
PaliGemma WaveUI 896: 0.49

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご