Uground V1 72B

🚀 UGround-V1-72B (Qwen2-VL-Based)(w/o LoRA)

UGround is a powerful GUI visual grounding model trained with a straightforward approach. For more details, visit our homepage and check out our paper. This project is a collaborative effort between OSUNLP and Orby AI.

• Homepage: UGround Homepage
• Repository: UGround Repository
• Paper: ArXiv Paper
• Demo: Hugging Face Demo
• Point of Contact: Boyu Gou

✨ Features

Models

• Model-V1:
  • Initial UGround
  • UGround-V1-2B (Qwen2-VL)
  • UGround-V1-7B (Qwen2-VL)
  • UGround-V1-72B (Qwen2-VL)
  • Training Data

Release Plan

• Model Weights:
  • Model Weights on Hugging Face
    • Initial Version (used in the paper)
    • Qwen2-VL-Based V1
      • 2B
      • 7B
      • 72B
• Code:
  • Inference Code of UGround (Initial & Qwen2-VL-Based)
  • Offline Experiments (Code, Results, and Useful Resources)
    • ScreenSpot (along with referring expressions generated by GPT-4/4o)
    • Multimodal-Mind2Web
    • OmniAct
    • Android Control
  • Online Experiments
    • Mind2Web-Live-SeeAct-V
    • AndroidWorld-SeeAct-V
  • Data Synthesis Pipeline (Coming Soon)
• Training-Data (V1)
• Online Demo (HF Spaces)

Main Results

GUI Visual Grounding: ScreenSpot (Standard Setting)

GUI Visual Grounding: ScreenSpot (Agent Setting)

📦 Installation

The code of Qwen2-VL has been in the latest Hugging face transformers. We advise you to build from source with the command pip install git+https://github.com/huggingface/transformers, or you might encounter the following error:

KeyError: 'qwen2_vl'

You can also install the toolkit for handling visual input using the following command:

pip install qwen-vl-utils

💻 Usage Examples

Inference

vLLM server

vllm serve osunlp/UGround-V1-7B  --api-key token-abc123 --dtype float16

or

python -m vllm.entrypoints.openai.api_server --served-model-name osunlp/UGround-V1-7B --model osunlp/UGround-V1-7B --dtype float16 

You can find more instruction about training and inference in Qwen2-VL's Official Repo.

Visual Grounding Prompt

def format_openai_template(description: str, base64_image):
    return [
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"},
                },
                {
                    "type": "text",
                    "text": f"""
  Your task is to help the user identify the precise coordinates (x, y) of a specific area/element/object on the screen based on a description.

  - Your response should aim to point to the center or a representative point within the described area/element/object as accurately as possible.
  - If the description is unclear or ambiguous, infer the most relevant area or element based on its likely context or purpose.
  - Your answer should be a single string (x, y) corresponding to the point of the interest.

  Description: {description}

  Answer:"""
                },
            ],
        },
    ]


messages = format_openai_template(description, base64_image)

completion = await client.chat.completions.create(
    model=args.model_path,
    messages=messages,
    temperature=0  # REMEMBER to set temperature to ZERO!
# REMEMBER to set temperature to ZERO!
# REMEMBER to set temperature to ZERO!
)

# The output will be in the range of [0,1000), which is compatible with the original Qwen2-VL
# So the actual coordinates should be (x/1000*width, y/1000*height)

Quickstart

from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info

# default: Load the model on the available device(s)
model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-72B-Instruct", torch_dtype="auto", device_map="auto"
)

# We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.

📚 Documentation

Qwen2-VL-72B-Instruct

Introduction

We're excited to unveil Qwen2-VL, the latest iteration of our Qwen-VL model, representing nearly a year of innovation.

What’s New in Qwen2-VL?

Key Enhancements

• SoTA understanding of images of various resolution & ratio: Qwen2-VL achieves state-of-the-art performance on visual understanding benchmarks, including MathVista, DocVQA, RealWorldQA, MTVQA, etc.
• Understanding videos of 20min+: Qwen2-VL can understand videos over 20 minutes for high-quality video-based question answering, dialog, content creation, etc.
• Agent that can operate your mobiles, robots, etc.: with the abilities of complex reasoning and decision making, Qwen2-VL can be integrated with devices like mobile phones, robots, etc., for automatic operation based on visual environment and text instructions.
• Multilingual Support: to serve global users, besides English and Chinese, Qwen2-VL now supports the understanding of texts in different languages inside images, including most European languages, Japanese, Korean, Arabic, Vietnamese, etc.

Model Architecture Updates

• Naive Dynamic Resolution: Unlike before, Qwen2-VL can handle arbitrary image resolutions, mapping them into a dynamic number of visual tokens, offering a more human-like visual processing experience.

• Multimodal Rotary Position Embedding (M-ROPE): Decomposes positional embedding into parts to capture 1D textual, 2D visual, and 3D video positional information, enhancing its multimodal processing capabilities.

We have three models with 2, 8 and 72 billion parameters. This repo contains the instruction-tuned 72B Qwen2-VL model. For more information, visit our Blog and GitHub.

Evaluation

Image Benchmarks

Video Benchmarks

Agent Benchmarks

SR, GC, TM and EM are short for success rate, goal-condition success, type match and exact match. ALFRED is supported by SAM^[6].

1. Self-Curated Function Call Benchmark by Qwen Team
2. Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
3. Android in the Zoo: Chain-of-Action-Thought for GUI Agents
4. ThinkBot: Embodied Instruction Following with Thought Chain Reasoning
5. MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation
6. Segment Anything.

Multilingual Benchmarks

🔧 Technical Details

The UGround model is a GUI visual grounding model based on Qwen2-VL. It is trained with a simple recipe and shows excellent performance on various benchmarks. The model architecture of Qwen2-VL has several key updates, such as Naive Dynamic Resolution and Multimodal Rotary Position Embedding (M-ROPE), which enhance its visual processing and multimodal understanding capabilities.

📄 License

This project is licensed under the tongyi-qianwen license.

📚 Citation Information

If you find this work useful, please consider citing our papers:

@article{gou2024uground,
        title={Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents},
        author={Boyu Gou and Ruohan Wang and Boyuan Zheng and Yanan Xie and Cheng Chang and Yiheng Shu and Huan Sun and Yu Su},
        journal={arXiv preprint arXiv:2410.05243},
        year={2024},
        url={https://arxiv.org/abs/2410.05243},
      }

@article{zheng2023seeact,
        title={GPT-4V(ision) is a Generalist Web Agent, if Grounded},
        author={Boyuan Zheng and Boyu Gou and Jihyung Kil and Huan Sun and Yu Su},
        journal={arXiv preprint arXiv:2401.01614},
        year={2024},
      }