Uground V1 7B

🚀 UGround-V1-7B (Qwen2-VL-Based)

UGround is a powerful GUI visual grounding model trained using a simple approach. For more detailed information, please visit our homepage and refer to our paper. This project is a collaborative effort between the OSU NLP Group and Orby AI.

• Homepage: https://osu-nlp-group.github.io/UGround/
• Repository: https://github.com/OSU-NLP-Group/UGround
• Paper (ICLR'25 Oral): https://arxiv.org/abs/2410.05243
• Demo: https://huggingface.co/spaces/orby-osu/UGround
• Point of Contact: Boyu Gou

✨ Features

• UGround is a strong GUI visual grounding model trained with a simple recipe.
• It is a collaboration between OSU NLP Group and Orby AI.

📦 Installation

No installation steps are provided in the original README.

💻 Usage Examples

Inference

vLLM server

vllm serve osunlp/UGround-V1-7B  --api-key token-abc123 --dtype float16

or

python -m vllm.entrypoints.openai.api_server --served-model-name osunlp/UGround-V1-7B --model osunlp/UGround-V1-7B --dtype float16 

You can find more instruction about training and inference in Qwen2-VL's Official Repo.

Visual Grounding Prompt

def format_openai_template(description: str, base64_image):
    return [
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"},
                },
                {
                    "type": "text",
                    "text": f"""
  Your task is to help the user identify the precise coordinates (x, y) of a specific area/element/object on the screen based on a description.

  - Your response should aim to point to the center or a representative point within the described area/element/object as accurately as possible.
  - If the description is unclear or ambiguous, infer the most relevant area or element based on its likely context or purpose.
  - Your answer should be a single string (x, y) corresponding to the point of the interest.

  Description: {description}

  Answer:"""
                },
            ],
        },
    ]


messages = format_openai_template(description, base64_image)

completion = await client.chat.completions.create(
    model=args.model_path,
    messages=messages,
    temperature=0  # REMEMBER to set temperature to ZERO!
# REMEMBER to set temperature to ZERO!
# REMEMBER to set temperature to ZERO!
)

# The output will be in the range of [0,1000), which is compatible with the original Qwen2-VL
# So the actual coordinates should be (x/1000*width, y/1000*height)

📚 Documentation

Models

• Model-V1:
  • Initial UGround:
  • UGround-V1-2B (Qwen2-VL)
  • UGround-V1-7B (Qwen2-VL)
  • UGround-V1-72B (Qwen2-VL)
  • Training Data

Release Plan

• [x] Model Weights
  • [x] Initial Version (the one used in the paper)
  • [x] Qwen2-VL-Based V1
    • [x] 2B
    • [x] 7B
    • [x] 72B
• [x] Code
  • [x] Inference Code of UGround (Initial & Qwen2-VL-Based
  • [x] Offline Experiments (Code, Results, and Useful Resources)
    • [x] ScreenSpot (along with referring expressions generated by GPT-4/4o)
    • [x] Multimodal-Mind2Web
    • [x] OmniAct
    • [x] Android Control
  • [x] Online Experiments
    • [x] Mind2Web-Live-SeeAct-V
    • [x] AndroidWorld-SeeAct-V
  • [ ] Data Synthesis Pipeline (Coming Soon)
• [x] Training-Data (V1)
• [x] Online Demo (HF Spaces)

Main Results

GUI Visual Grounding: ScreenSpot (Standard Setting)

GUI Visual Grounding: ScreenSpot (Agent Setting)

🔧 Technical Details

No technical details are provided in the original README.

📄 License

This project is licensed under the Apache-2.0 license.

Citation Information

If you find this work useful, please consider citing our papers:

@article{gou2024uground,
        title={Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents},
        author={Boyu Gou and Ruohan Wang and Boyuan Zheng and Yanan Xie and Cheng Chang and Yiheng Shu and Huan Sun and Yu Su},
        journal={arXiv preprint arXiv:2410.05243},
        year={2024},
        url={https://arxiv.org/abs/2410.05243},
      }

@article{zheng2023seeact,
        title={GPT-4V(ision) is a Generalist Web Agent, if Grounded},
        author={Boyuan Zheng and Boyu Gou and Jihyung Kil and Huan Sun and Yu Su},
        journal={arXiv preprint arXiv:2401.01614},
        year={2024},
      }