Eagle2-2B開源視覺語言模型 - 輕量級設計，高效快速處理視覺語言任務

首頁

Eagle2 2B

由nvidia開發

Eagle2是NVIDIA推出的高性能視覺語言模型家族，專注於通過數據策略和訓練方案提升開源視覺語言模型的性能。Eagle2-2B是該系列中的輕量級模型，在保持強勁性能的同時實現卓越效率和速度。

文本生成圖像

Transformers

其他#多模態視覺語言 #高效輕量級 #多語言支持

下載量 667

發布時間 : 1/10/2025

模型概述

Eagle2-2B是一個融合視覺和語言能力的多模態模型，能夠處理圖像、文本和視頻輸入，執行多種視覺語言任務。

模型特點

高效輕量

在2B參數規模下實現卓越的性能和速度平衡

多模態處理

支持圖像、文本和視頻輸入的綜合處理能力

長上下文支持

支持長達16K tokens的上下文長度

高性能基準

在多個視覺語言基準測試中表現優異

模型能力

圖像理解與描述

視覺問答

文檔理解

圖表分析

視頻內容理解

多模態推理

使用案例

文檔處理

文檔問答

從掃描文檔或PDF中提取信息並回答問題

在DocVQA測試集上達到88.0分

視覺問答

圖像內容問答

回答關於圖像內容的複雜問題

在TextVQA驗證集上達到79.1分

教育輔助

圖表理解

解釋和分析各種圖表數據

在ChartQA測試集上達到82.0分

🚀 Eagle-2

我們很高興地推出最新的Eagle2系列視覺語言模型。該模型在多語言處理上表現出色，結合了多個基礎模型的優勢，具有高效、高性能的特點，為開源視覺語言模型的發展提供了新的思路和解決方案。

[📂 GitHub] [📜 Eagle2技術報告] [🤗 HF演示]

🚀 快速開始

安裝依賴

pip install transformers
pip install flash-attn

單張圖像輸入示例

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://www.ilankelman.org/stopsigns/australia.jpg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)]
image_inputs, video_inputs = processor.process_vision_info(messages)
inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True)
inputs = inputs.to("cuda")
model = model.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

流式生成示例

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel, AutoTokenizer
import torch

from transformers import TextIteratorStreamer
import threading


model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, attn_implementation='flash_attention_2', torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://www.ilankelman.org/stopsigns/australia.jpg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)]
image_inputs, video_inputs = processor.process_vision_info(messages)
inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True)
inputs = inputs.to("cuda")
model = model.to("cuda")

streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

generation_kwargs = dict(
    **inputs,
    streamer=streamer,
    max_new_tokens=1024,
    do_sample=True,
    top_p=0.95,
    temperature=0.8
)
thread = threading.Thread(target=model.generate, kwargs=generation_kwargs)
thread.start()


for new_text in streamer:
    print(new_text, end="", flush=True)

多張圖像輸入示例

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://www.ilankelman.org/stopsigns/australia.jpg",
            },
            {
                "type": "image",
                "image": "https://www.nvidia.com/content/dam/en-zz/Solutions/about-nvidia/logo-and-brand/01-nvidia-logo-vert-500x200-2c50-d@2x.png",
            },
            {"type": "text", "text": "Describe these two images."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)]
image_inputs, video_inputs = processor.process_vision_info(messages)
inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True)
inputs = inputs.to("cuda")
model = model.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

單視頻輸入示例

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "video": "../Eagle2-8B/space_woaudio.mp4",
            },
            {"type": "text", "text": "Describe this video."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)]
image_inputs, video_inputs, video_kwargs = processor.process_vision_info(messages, return_video_kwargs=True)

inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True, videos_kwargs=video_kwargs)
inputs = inputs.to("cuda")
model = model.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

多視頻輸入示例

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "video": "../Eagle2-8B/space_woaudio.mp4",
                "nframes": 10,
            },
            {
                "type": "video",
                "video": "../Eagle2-8B/video_ocr.mp4",
                "nframes": 10,
            },
            {"type": "text", "text": "Describe these two videos respectively."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)]
image_inputs, video_inputs, video_kwargs = processor.process_vision_info(messages, return_video_kwargs=True)
inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True, videos_kwargs=video_kwargs)
inputs = inputs.to("cuda")
model = model.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

批量推理示例

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages1 = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://www.ilankelman.org/stopsigns/australia.jpg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

messages2 = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://www.nvidia.com/content/dam/en-zz/Solutions/about-nvidia/logo-and-brand/01-nvidia-logo-vert-500x200-2c50-d@2x.png",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
) for messages in [messages1, messages2]]
image_inputs, video_inputs = processor.process_vision_info([messages1, messages2])
inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True)
inputs = inputs.to("cuda")
model = model.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

✨ 主要特性

模型架構更新：將模型架構更新為 eagle_2_5_vl，以支持 generate 功能。
多語言支持：支持多種語言，具有廣泛的適用性。
多模態輸入：支持純文本、單圖像、多圖像、單視頻、多視頻等多種輸入類型。

📦 模型庫

我們提供以下模型：

模型名稱	大語言模型	視覺模型	最大長度	Hugging Face鏈接
Eagle2-1B	Qwen2.5-0.5B-Instruct	Siglip	16K	🤗 鏈接
Eagle2-2B	Qwen2.5-1.5B-Instruct	Siglip	16K	🤗 鏈接
Eagle2-9B	Qwen2.5-7B-Instruct	Siglip+ConvNext	16K	🤗 鏈接

📊 基準測試結果

基準測試	InternVL2-2B	InternVL2.5-2B	InternVL2-4B	Qwen2-VL-2B	Eagle2-2B
DocVQA_測試集	86.9	88.7	89.2	90.1	88.0
ChartQA_測試集	76.2	79.2	81.5	73.0	82.0
InfoVQA_測試集	58.9	60.9	67.0	65.5	65.8
TextVQA_驗證集	73.4	74.3	74.4	79.7	79.1
OCRBench	784	804	788	809	818
MME_總和	1876.8	2138.2	2059.8	1872.0	2109.8
RealWorldQA	57.3	60.1	60.7	62.6	63.1
AI2D_測試集	74.1	74.9	74.7	78.9	79.3
MMMU_驗證集	36.3	43.6	47.9	41.1	43.1
MMVet_GPT-4-Turbo	39.5	60.8	51.0	49.5	53.8
HallBench_平均值	37.9	42.6	41.9	41.7	45.8
MathVista_測試小集	46.3	51.3	58.6	43.0	54.7
MMstar	50.1	53.7	54.3	48.0	56.4

📋 待辦事項

[ ] 支持 vLLM 推理
[ ] 提供 AWQ 量化權重
[ ] 提供微調腳本

📄 許可證

代碼根據 Apache 2.0 許可證發佈。
預訓練模型權重根據知識共享署名-非商業性使用 4.0 國際許可協議發佈。
該服務僅供非商業研究預覽使用，並受以下許可證和條款約束：
- Qwen2.5-1.5B-Instruct 模型許可證：Apache-2.0
- PaliGemma 模型許可證：Gemma 許可證