library_name: ultralytics pipeline_tag: object-detection license: agpl-3.0

YOLOE: Real-Time Seeing Anything

Official PyTorch implementation of YOLOE.

Comparison of performance, training cost, and inference efficiency between YOLOE (Ours) and YOLO-Worldv2 in terms of open text prompts.

YOLOE: Real-Time Seeing Anything.
Ao Wang*, Lihao Liu*, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding

We introduce YOLOE(ye), a highly efficient, unified, and open object detection and segmentation model, like human eye, under different prompt mechanisms, like texts, visual inputs, and prompt-free paradigm.

Performance

Zero-shot detection evaluation

• Fixed AP is reported on LVIS minival set with text (T) / visual (V) prompts.
• Training time is for text prompts with detection based on 8 Nvidia RTX4090 GPUs.
• FPS is measured on T4 with TensorRT and iPhone 12 with CoreML, respectively.
• For training data, OG denotes Objects365v1 and GoldG.
• YOLOE can become YOLOs after re-parameterization with zero inference and transferring overhead.

Zero-shot segmentation evaluation

• The model is the same as above in Zero-shot detection evaluation.
• Standard AP^m is reported on LVIS val set with text (T) / visual (V) prompts.

Prompt-free evaluation

• The model is the same as above in Zero-shot detection evaluation except the specialized prompt embedding.
• Fixed AP is reported on LVIS minival set and FPS is measured on Nvidia T4 GPU with Pytorch.

Downstream transfer on COCO

• During transferring, YOLOE-v8 / YOLOE-11 is exactly the same as YOLOv8 / YOLO11.
• For Linear probing, only the last conv in classification head is trainable.
• For Full tuning, all parameters are trainable.

Installation

conda virtual environment is recommended.

conda create -n yoloe python=3.10 -y
conda activate yoloe

pip install -r requirements.txt
pip install -e .
pip install -e lvis-api
pip install -e ml-mobileclip
pip install -e CLIP

Demo

If desired objects are not identified, pleaset set a smaller confidence threshold, e.g., for visual prompts with handcrafted shape or cross-image prompts.

# Optional for mirror: export HF_ENDPOINT=https://hf-mirror.com
pip install gradio==4.42.0 gradio_image_prompter==0.1.0 fastapi==0.112.2
python app.py
# Please visit http://127.0.0.1:7860

Prediction

Text prompt

python predict.py

Visual prompt

python predict_vp.py

Prompt free

python predict_pf.py

Validation

Data

• Please download LVIS following here or lvis.yaml.
• We use this minival.txt with background images for evaluation.

# For evaluation with visual prompt, please obtain the referring data.
python tools/generate_lvis_visual_prompt_data.py

Zero-shot evaluation on LVIS

• For text prompts, python val.py.
• For visual prompts, python val_vp.py

For Fixed AP, please refer to the comments in val.py and val_vp.py, and use tools/eval_fixed_ap.py for evaluation.

Prompt-free evaluation

python val_pe_free.py
python tools/eval_open_ended.py --json ../datasets/lvis/annotations/lvis_v1_minival.json --pred runs/detect/val/predictions.json --fixed

Downstream transfer on COCO

python val_coco.py

Training

The training includes three stages:

• YOLOE is trained with text prompts for detection and segmentation for 30 epochs.
• Only visual prompt encoder (SAVPE) is trained with visual prompts for 2 epochs.
• Only specialized prompt embedding for prompt free is trained for 1 epochs.

Data

For annotations, you can directly use our preprocessed ones or use the following script to obtain the processed annotations with segmentation masks.

# Generate segmentation data
conda create -n sam2 python==3.10.16
conda activate sam2
pip install -r sam2/requirements.txt
pip install -e sam2/

python tools/generate_sam_masks.py --img-path ../datasets/Objects365v1/images/train --json-path ../datasets/Objects365v1/annotations/objects365_train.json --batch
python tools/generate_sam_masks.py --img-path ../datasets/flickr/full_images/ --json-path ../datasets/flickr/annotations/final_flickr_separateGT_train.json
python tools/generate_sam_masks.py --img-path ../datasets/mixed_grounding/gqa/images --json-path ../datasets/mixed_grounding/annotations/final_mixed_train_no_coco.json

# Generate objects365v1 labels
python tools/generate_objects365v1.py

Then, please generate the data and embedding cache for training.

# Generate grounding segmentation cache
python tools/generate_grounding_cache.py --img-path ../datasets/flickr/full_images/ --json-path ../datasets/flickr/annotations/final_flickr_separateGT_train_segm.json
python tools/generate_grounding_cache.py --img-path ../datasets/mixed_grounding/gqa/images --json-path ../datasets/mixed_grounding/annotations/final_mixed_train_no_coco_segm.json

# Generate train label embeddings
python tools/generate_label_embedding.py
python tools/generate_global_neg_cat.py

At last, please download MobileCLIP-B(LT) for text encoder.

wget https://docs-assets.developer.apple.com/ml-research/datasets/mobileclip/mobileclip_blt.pt

Text prompt

# For models with l scale, please change the initialization by referring to the comments in Line 549 in ultralytics/nn/moduels/head.py
# If you want to train YOLOE only for detection, you can use `train.py` 
python train_seg.py

Visual prompt

# For visual prompt, because only SAVPE is trained, we can adopt the detection pipleline with less training time

# First, obtain the detection model
python tools/convert_segm2det.py
# Then, train the SAVPE module
python train_vp.py
# After training, please use tools/get_vp_segm.py to add the segmentation head
# python tools/get_vp_segm.py

Prompt free

# Generate LVIS with single class for evaluation during training
python tools/generate_lvis_sc.py

# Similar to visual prompt, because only the specialized prompt embedding is trained, we can adopt the detection pipleline with less training time
python tools/convert_segm2det.py
python train_pe_free.py
# After training, please use tools/get_pf_free_segm.py to add the segmentation head
# python tools/get_pf_free_segm.py

Transferring

After pretraining, YOLOE-v8 / YOLOE-11 can be re-parameterized into the same architecture as YOLOv8 / YOLO11, with zero overhead for transferring.

Linear probing

Only the last conv, ie., the prompt embedding, is trainable.

python train_pe.py

Full tuning

All parameters are trainable, for better performance.

# For models with s scale, please change the epochs to 160 for longer training
python train_pe_all.py

Export

After re-parameterization, YOLOE-v8 / YOLOE-11 can be exported into the identical format as YOLOv8 / YOLO11.

pip install onnx coremltools onnxslim
python export.py

Benchmark

• For TensorRT, please refer to benchmark.sh.
• For CoreML, please use the benchmark tool from XCode 14.
• For prompt-free setting, please refer to tools/benchmark_pf.py.

Acknowledgement

The code base is built with ultralytics, YOLO-World, MobileCLIP, lvis-api, CLIP, and GenerateU.

Thanks for the great implementations!

Citation

If our code or models help your work, please cite our paper:

@misc{wang2025yoloerealtimeseeing,
      title={YOLOE: Real-Time Seeing Anything}, 
      author={Ao Wang and Lihao Liu and Hui Chen and Zijia Lin and Jungong Han and Guiguang Ding},
      year={2025},
      eprint={2503.07465},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2503.07465}, 
}