Paligemma-3b-ft-waveui-896 Open Source Model - Free for Precise Detection of UI Elements

Paligemma 3b Ft Waveui 896

Developed by agentsea

A UI element detection model fine-tuned from PaliGemma 3B 896-resolution weights, specializing in object detection tasks

Image-to-Text

Transformers

English#UI Element Detection #High-Precision Positioning #Agent Development

Downloads 43

Release Time : 7/24/2024

Model Overview

This model is fine-tuned on the WaveUI dataset and excels at UI element detection, serving as a key component in building agent plans

Model Features

High-Precision UI Element Detection

Achieves an average IoU of 0.49 on the test set, significantly outperforming mainstream closed-source models

Optimized for WaveUI Dataset

Trained specifically on a dataset of approximately 80,000 annotated UI elements

896 High-Resolution Support

Fine-tuned from the 896-resolution PaliGemma model, suitable for high-precision detection needs

Model Capabilities

UI Element Detection

Object Detection

Vision-Language Understanding

Use Cases

Agent Development

UI Automation Testing

Automatically identifies and locates UI elements in application interfaces

Improves automation efficiency and accuracy in testing

Intelligent Interaction Agent

Provides interface element recognition capabilities for agents

Enhances agent interaction with graphical interfaces

🚀 Paligemma WaveUI

This project fine - tunes the PaliGemma 3B 896 - res weights on the WaveUI dataset for object - detection, contributing to the open - source toolkit for building agents at AgentSea.

🚀 Quick Start

To start using this model, run the following:

from transformers import AutoProcessor, PaliGemmaForConditionalGeneration

model = PaliGemmaForConditionalGeneration.from_pretrained("agentsea/paligemma-3b-ft-waveui-896").eval()
processor = AutoProcessor.from_pretrained("agentsea/paligemma-3b-ft-waveui-896")

✨ Features

This fine - tune is based on the Paligemma 896 model, using the WaveUI dataset with about 80k examples of labeled UI elements.
It is designed for the object detection task, specifically targeting UI element detection to support the open - source toolkit at AgentSea.
You can find a demo for this model here.

📚 Documentation

Model Details

Model Description

This fine - tune was done atop of the Paligemma 896 model, using the WaveUI dataset, which contains ~80k examples of labeled UI elements. The fine - tune was done for the object detection task. Specifically, this model aims to perform well at UI element detection, as part of a wider effort to enable our open - source toolkit for building agents at AgentSea.

Developed by: https://agentsea.ai/
Language(s) (NLP): en
Finetuned from model: https://huggingface.co/google/paligemma-3b-pt-896

Notes

The only task used in the fine - tune was the object detection task, so it might not perform well in other types of tasks.

Data

We used the WaveUI dataset for this fine - tune. Before using it, we preprocessed the data to use the Paligemma bounding - box format.

Evaluation

We calculated the mean IoU over 1024 examples of the test set using 3 different closed - source models: Gemini 1.5 Pro, Claude 3.5 Sonnet and GPT 4o. We also ran this same calculation using the PaliGemma WaveUI fine - tunes. We obtained the following values:

Gemini 1.5 Pro: 0.12
Claude 3.5 Sonnet: 0.05
GPT 4o: 0.05
PaliGemma Widgetcap+WaveUI 448: 0.40
PaliGemma WaveUI 896: 0.49

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご