開源Eagle2-1B視覺語言模型 - 助力數據透明化開發高性能模型

首頁

Eagle2 1B

由nvidia開發

鷹隼2號是一個高性能視覺語言模型家族，專注於數據策略和訓練方案的透明性，旨在推動開源社區開發具有競爭力的視覺語言模型。

圖像生成文本

Transformers

其他#多模態視覺語言 #高效小參數 #文檔圖像理解

下載量 1,791

發布時間 : 1/10/2025

模型概述

鷹隼2號-1B是一個緊湊高效的視覺語言模型，專為需要快速推理和最小計算資源的場景設計，同時保持核心性能。

模型特點

高效性能

在緊湊的1B參數規模下，仍能保持與更大模型競爭的性能。

多語言支持

支持多種語言的視覺語言理解任務。

長上下文處理

支持長達16K的上下文長度，適合處理複雜任務。

透明數據策略

強調數據策略的透明性，便於復現和創新。

模型能力

視覺語言理解

圖像文本生成

多模態推理

文檔理解

圖表分析

使用案例

文檔處理

DocVQA

文檔視覺問答

測試集準確率81.8

圖表分析

ChartQA

圖表問答

測試集準確率77.0

視覺問答

TextVQA

文本視覺問答

驗證集準確率76.6

🚀 Eagle-2

我們很高興地宣佈推出最新的Eagle2系列視覺語言模型（VLM）。該模型聚焦於從以數據為中心的視角進行VLM的後訓練，分享瞭如何從頭構建有效數據策略的見解。結合強大的訓練方法和模型設計，我們推出了性能出色的Eagle2系列模型，旨在助力開源社區以透明的流程開發具有競爭力的VLM。

[📂 GitHub] [📜 Eagle2 Tech Report] [🤗 HF Demo]

🆕 最新消息

我們將模型架構更新為 eagle_2_5_vl，以支持 generate 功能。

✨ 主要特性

多語言支持：支持多種語言，具有廣泛的適用性。
多種輸入類型：支持純文本輸入、單圖像輸入、多圖像輸入、視頻輸入。
模型架構更新：更新為 eagle_2_5_vl 架構，支持 generate 功能。

📦 安裝指南

安裝依賴項

pip install transformers
pip install flash-attn

💻 使用示例

基礎用法

單圖像輸入

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://www.ilankelman.org/stopsigns/australia.jpg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)]
image_inputs, video_inputs = processor.process_vision_info(messages)
inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True)
inputs = inputs.to("cuda")
model = model.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

高級用法

流式生成

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel, AutoTokenizer
import torch

from transformers import TextIteratorStreamer
import threading


model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, attn_implementation='flash_attention_2', torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://www.ilankelman.org/stopsigns/australia.jpg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)]
image_inputs, video_inputs = processor.process_vision_info(messages)
inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True)
inputs = inputs.to("cuda")
model = model.to("cuda")

streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

generation_kwargs = dict(
    **inputs,
    streamer=streamer,
    max_new_tokens=1024,
    do_sample=True,
    top_p=0.95,
    temperature=0.8
)
thread = threading.Thread(target=model.generate, kwargs=generation_kwargs)
thread.start()


for new_text in streamer:
    print(new_text, end="", flush=True)

多圖像輸入

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://www.ilankelman.org/stopsigns/australia.jpg",
            },
            {
                "type": "image",
                "image": "https://www.nvidia.com/content/dam/en-zz/Solutions/about-nvidia/logo-and-brand/01-nvidia-logo-vert-500x200-2c50-d@2x.png",
            },
            {"type": "text", "text": "Describe these two images."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)]
image_inputs, video_inputs = processor.process_vision_info(messages)
inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True)
inputs = inputs.to("cuda")
model = model.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

單視頻輸入

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "video": "../Eagle2-8B/space_woaudio.mp4",
            },
            {"type": "text", "text": "Describe this video."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)]
image_inputs, video_inputs, video_kwargs = processor.process_vision_info(messages, return_video_kwargs=True)

inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True, videos_kwargs=video_kwargs)
inputs = inputs.to("cuda")
model = model.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

多視頻輸入

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "video": "../Eagle2-8B/space_woaudio.mp4",
                "nframes": 10,
            },
            {
                "type": "video",
                "video": "../Eagle2-8B/video_ocr.mp4",
                "nframes": 10,
            },
            {"type": "text", "text": "Describe these two videos respectively."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)]
image_inputs, video_inputs, video_kwargs = processor.process_vision_info(messages, return_video_kwargs=True)
inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True, videos_kwargs=video_kwargs)
inputs = inputs.to("cuda")
model = model.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

批量推理

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages1 = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://www.ilankelman.org/stopsigns/australia.jpg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

messages2 = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://www.nvidia.com/content/dam/en-zz/Solutions/about-nvidia/logo-and-brand/01-nvidia-logo-vert-500x200-2c50-d@2x.png",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
) for messages in [messages1, messages2]]
image_inputs, video_inputs = processor.process_vision_info([messages1, messages2])
inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True)
inputs = inputs.to("cuda")
model = model.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

📚 詳細文檔

模型庫

我們提供以下模型：

模型名稱	大語言模型（LLM）	視覺模型	最大長度	Hugging Face鏈接
Eagle2-1B	Qwen2.5-0.5B-Instruct	Siglip	16K	🤗 鏈接
Eagle2-2B	Qwen2.5-1.5B-Instruct	Siglip	16K	🤗 鏈接
Eagle2-9B	Qwen2.5-7B-Instruct	Siglip+ConvNext	16K	🤗 鏈接

基準測試結果

基準測試	LLaVa-One-Vision-0.5B	InternVL2-1B	InternVL2.5-1B	Qwen2-VL-2B	Eagle2-1B
DocVQA_test	70.0	81.7	84.8	90.1	81.8
ChartQA_test	61.4	72.9	75.9	73.0	77.0
InfoVQA_test	41.8	50.9	56.0	65.5	54.8
TextVQA_val	-	70.0	72.0	79.7	76.6
OCRBench	565	754	785	809	767
MME_sum	1438.0	1794.4	1950.5	1872.0	1790.2
RealWorldQA	55.6	50.3	57.5	62.6	55.4
AI2D_test	57.1	64.1	69.3	74.7	70.9
MMMU_val	31.4	36.7	40.9	41.1	38.8
MMVet_{GPT - 4 - Turbo}	32.2	32.7	48.8	49.5	40.9
HallBench_avg	27.9	34.0	39.0	41.7	35.3
MathVista_testmini	33.8	37.7	43.2	43.0	45.3
MMstar	37.7	45.7	50.1	48.0	48.5

📋 待辦事項

[ ] 支持vLLM推理
[ ] 提供AWQ量化權重
[ ] 提供微調腳本

📄 許可證

代碼根據 LICENSE 文件中的Apache 2.0許可證發佈。
預訓練模型權重根據 Creative Commons Attribution: Non - Commercial 4.0 International 許可證發佈。
該服務僅用於非商業研究預覽，並受以下許可證和條款約束：
- Qwen2.5-0.5B-Instruct的模型許可證：Apache - 2.0
- PaliGemma的模型許可證：Gemma license