Ferret UI Llama8b

🚀 Ferret - UI

Ferret - UIは、参照、接地、推論タスク用に設計された最初のUI中心のマルチモーダル大規模言語モデル（MLLM）です。Gemma - 2BとLlama - 3 - 8Bをベースに構築されており、複雑なUIタスクを実行することができます。これはferret - UIのLlama - 3 - 8Bバージョンです。Appleによるこの論文に基づいています。

🚀 クイックスタート

まず、builder.py、conversation.py、inference.py、model_UI.py、mm_utils.pyをローカルにダウンロードする必要があります。

wget https://huggingface.co/jadechoghari/Ferret-UI-Gemma2b/raw/main/conversation.py
wget https://huggingface.co/jadechoghari/Ferret-UI-Gemma2b/raw/main/builder.py
wget https://huggingface.co/jadechoghari/Ferret-UI-Gemma2b/raw/main/inference.py
wget https://huggingface.co/jadechoghari/Ferret-UI-Gemma2b/raw/main/model_UI.py
wget https://huggingface.co/jadechoghari/Ferret-UI-Gemma2b/raw/main/mm_utils.py

💻 使用例

基本的な使用法

from inference import inference_and_run
image_path = "appstore_reminders.png"
prompt = "Describe the image in details"

# Call the function without a box
inference_text = inference_and_run(image_path, prompt)

print("Inference Text:", inference_text)

高度な使用法

# Task with bounding boxes
image_path = "appstore_reminders.png"
prompt = "What's inside the selected region?"
box = [189, 906, 404, 970]

inference_text = inference_and_run(
    image_path=image_path, 
    prompt=prompt, 
    conv_mode="ferret_llama_3", 
    model_path="jadechoghari/Ferret-UI-Llama8b", 
    box=box
)

print("Inference Text:", inference_text)

# GROUNDING PROMPTS
GROUNDING_TEMPLATES = [
	'\nProvide the bounding boxes of the mentioned objects.',
 	'\nInclude the coordinates for each mentioned object.',
	'\nLocate the objects with their coordinates.',
	'\nAnswer in [x1, y1, x2, y2] format.',
	'\nMention the objects and their locations using the format [x1, y1, x2, y2].',
	'\nDraw boxes around the mentioned objects.',
	'\nUse boxes to show where each thing is.',
	'\nTell me where the objects are with coordinates.',
	'\nList where each object is with boxes.',
	'\nShow me the regions with boxes.'
]