Ferret-UI-Gemma2b: An Open-Source Multimodal Large Model to Support UI Reference, Localization, and Inference Tasks

Ferret UI Gemma2b

Developed by jadechoghari

Ferret-UI is the first multimodal large language model focused on user interfaces, built on Gemma-2B, specifically designed for UI referencing, localization, and reasoning tasks.

Image-to-Text

Transformers

#UI Multimodal Understanding #UI Element Localization #Screen Content Reasoning

Downloads 302

Release Time : 10/9/2024

Model Overview

Ferret-UI is a multimodal large language model specializing in understanding and analyzing user interfaces (UI), capable of performing complex UI tasks such as referencing, localization, and reasoning.

Model Features

UI-Specific Multimodal Model

The first multimodal large language model dedicated to user interface understanding

Precise Localization Capability

Can accurately locate UI elements and provide bounding box coordinates

Complex Reasoning Ability

Capable of performing complex UI-related reasoning tasks

Model Capabilities

UI Element Recognition

UI Element Localization

UI Interface Description

UI Element Interaction Analysis

UI Layout Understanding

Use Cases

Mobile App Interface Analysis

App Interface Element Identification

Identify and describe various elements in mobile app interfaces

Accurately recognize UI components such as buttons, text fields, etc.

Interface Navigation Analysis

Analyze the navigation structure and flow of app interfaces

Understand the transition relationships between interfaces and user operation paths

UI Automated Testing

UI Element Verification

Verify the existence and position of UI elements

Ensure interface elements are presented according to design specifications

🚀 Ferret-UI (Gemma-2B Version)

Ferret-UI is the first UI-centric multimodal large language model (MLLM) tailored for referring, grounding, and reasoning tasks. Built upon Gemma-2B and Llama-3-8B, it can handle complex UI tasks. This is the Gemma-2B version of Ferret-UI, following this paper by Apple.

🚀 Quick Start

📦 Installation

You need to download builder.py, conversation.py, inference.py, model_UI.py, and mm_utils.py locally first.

wget https://huggingface.co/jadechoghari/Ferret-UI-Gemma2b/raw/main/conversation.py
wget https://huggingface.co/jadechoghari/Ferret-UI-Gemma2b/raw/main/builder.py
wget https://huggingface.co/jadechoghari/Ferret-UI-Gemma2b/raw/main/inference.py
wget https://huggingface.co/jadechoghari/Ferret-UI-Gemma2b/raw/main/model_UI.py
wget https://huggingface.co/jadechoghari/Ferret-UI-Gemma2b/raw/main/mm_utils.py

💻 Usage Examples

Basic Usage

from inference import inference_and_run
image_path = "appstore_reminders.png"
prompt = "Describe the image in details"

# Call the function without a box
inference_text = inference_and_run(image_path, prompt, conv_mode="ferret_gemma_instruct", model_path="jadechoghari/Ferret-UI-Gemma2b")

# Output processed text
print("Inference Text:", inference_text)

Advanced Usage

Task with bounding boxes

# Task with bounding boxes
image_path = "appstore_reminders.png"
prompt = "What's inside the selected region?"
box = [189, 906, 404, 970]

inference_text = inference_and_run(
    image_path=image_path, 
    prompt=prompt, 
    conv_mode="ferret_gemma_instruct", 
    model_path="jadechoghari/Ferret-UI-Gemma2b", 
    box=box
)
# you could also pass process_image=True
# to output: processed_image, inference_text = inference_and_run(...., process_image=True)

print("Inference Text:", inference_text)

GROUNDING PROMPTS

# GROUNDING PROMPTS
GROUNDING_TEMPLATES = [
    '\nProvide the bounding boxes of the mentioned objects.',
    '\nInclude the coordinates for each mentioned object.',
    '\nLocate the objects with their coordinates.',
    '\nAnswer in [x1, y1, x2, y2] format.',
    '\nMention the objects and their locations using the format [x1, y1, x2, y2].',
    '\nDraw boxes around the mentioned objects.',
    '\nUse boxes to show where each thing is.',
    '\nTell me where the objects are with coordinates.',
    '\nList where each object is with boxes.',
    '\nShow me the regions with boxes.'
]

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご