Eagle2-2B开源视觉语言模型 - 轻量级设计，高效快速处理视觉语言任务

首页

Eagle2 2B

由 nvidia 开发

Eagle2是NVIDIA推出的高性能视觉语言模型家族，专注于通过数据策略和训练方案提升开源视觉语言模型的性能。Eagle2-2B是该系列中的轻量级模型，在保持强劲性能的同时实现卓越效率和速度。

文本生成图像

Transformers

其他#多模态视觉语言 #高效轻量级 #多语言支持

下载量 667

发布时间 : 1/10/2025

模型简介

Eagle2-2B是一个融合视觉和语言能力的多模态模型，能够处理图像、文本和视频输入，执行多种视觉语言任务。

模型特点

高效轻量

在2B参数规模下实现卓越的性能和速度平衡

多模态处理

支持图像、文本和视频输入的综合处理能力

长上下文支持

支持长达16K tokens的上下文长度

高性能基准

在多个视觉语言基准测试中表现优异

模型能力

图像理解与描述

视觉问答

文档理解

图表分析

视频内容理解

多模态推理

使用案例

文档处理

文档问答

从扫描文档或PDF中提取信息并回答问题

在DocVQA测试集上达到88.0分

视觉问答

图像内容问答

回答关于图像内容的复杂问题

在TextVQA验证集上达到79.1分

教育辅助

图表理解

解释和分析各种图表数据

在ChartQA测试集上达到82.0分

🚀 Eagle-2

我们很高兴地推出最新的Eagle2系列视觉语言模型。该模型在多语言处理上表现出色，结合了多个基础模型的优势，具有高效、高性能的特点，为开源视觉语言模型的发展提供了新的思路和解决方案。

[📂 GitHub] [📜 Eagle2技术报告] [🤗 HF演示]

🚀 快速开始

安装依赖

pip install transformers
pip install flash-attn

单张图像输入示例

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://www.ilankelman.org/stopsigns/australia.jpg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)]
image_inputs, video_inputs = processor.process_vision_info(messages)
inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True)
inputs = inputs.to("cuda")
model = model.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

流式生成示例

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel, AutoTokenizer
import torch

from transformers import TextIteratorStreamer
import threading


model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, attn_implementation='flash_attention_2', torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://www.ilankelman.org/stopsigns/australia.jpg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)]
image_inputs, video_inputs = processor.process_vision_info(messages)
inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True)
inputs = inputs.to("cuda")
model = model.to("cuda")

streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

generation_kwargs = dict(
    **inputs,
    streamer=streamer,
    max_new_tokens=1024,
    do_sample=True,
    top_p=0.95,
    temperature=0.8
)
thread = threading.Thread(target=model.generate, kwargs=generation_kwargs)
thread.start()


for new_text in streamer:
    print(new_text, end="", flush=True)

多张图像输入示例

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://www.ilankelman.org/stopsigns/australia.jpg",
            },
            {
                "type": "image",
                "image": "https://www.nvidia.com/content/dam/en-zz/Solutions/about-nvidia/logo-and-brand/01-nvidia-logo-vert-500x200-2c50-d@2x.png",
            },
            {"type": "text", "text": "Describe these two images."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)]
image_inputs, video_inputs = processor.process_vision_info(messages)
inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True)
inputs = inputs.to("cuda")
model = model.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

单视频输入示例

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "video": "../Eagle2-8B/space_woaudio.mp4",
            },
            {"type": "text", "text": "Describe this video."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)]
image_inputs, video_inputs, video_kwargs = processor.process_vision_info(messages, return_video_kwargs=True)

inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True, videos_kwargs=video_kwargs)
inputs = inputs.to("cuda")
model = model.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

多视频输入示例

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "video": "../Eagle2-8B/space_woaudio.mp4",
                "nframes": 10,
            },
            {
                "type": "video",
                "video": "../Eagle2-8B/video_ocr.mp4",
                "nframes": 10,
            },
            {"type": "text", "text": "Describe these two videos respectively."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)]
image_inputs, video_inputs, video_kwargs = processor.process_vision_info(messages, return_video_kwargs=True)
inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True, videos_kwargs=video_kwargs)
inputs = inputs.to("cuda")
model = model.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

批量推理示例

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("nvidia/Eagle2-1B",trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("nvidia/Eagle2-1B", trust_remote_code=True, use_fast=True)
processor.tokenizer.padding_side = "left"

messages1 = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://www.ilankelman.org/stopsigns/australia.jpg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

messages2 = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://www.nvidia.com/content/dam/en-zz/Solutions/about-nvidia/logo-and-brand/01-nvidia-logo-vert-500x200-2c50-d@2x.png",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

text_list = [processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
) for messages in [messages1, messages2]]
image_inputs, video_inputs = processor.process_vision_info([messages1, messages2])
inputs = processor(text = text_list, images=image_inputs, videos=video_inputs, return_tensors="pt", padding=True)
inputs = inputs.to("cuda")
model = model.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

✨ 主要特性

模型架构更新：将模型架构更新为 eagle_2_5_vl，以支持 generate 功能。
多语言支持：支持多种语言，具有广泛的适用性。
多模态输入：支持纯文本、单图像、多图像、单视频、多视频等多种输入类型。

📦 模型库

我们提供以下模型：

模型名称	大语言模型	视觉模型	最大长度	Hugging Face链接
Eagle2-1B	Qwen2.5-0.5B-Instruct	Siglip	16K	🤗 链接
Eagle2-2B	Qwen2.5-1.5B-Instruct	Siglip	16K	🤗 链接
Eagle2-9B	Qwen2.5-7B-Instruct	Siglip+ConvNext	16K	🤗 链接

📊 基准测试结果

基准测试	InternVL2-2B	InternVL2.5-2B	InternVL2-4B	Qwen2-VL-2B	Eagle2-2B
DocVQA_测试集	86.9	88.7	89.2	90.1	88.0
ChartQA_测试集	76.2	79.2	81.5	73.0	82.0
InfoVQA_测试集	58.9	60.9	67.0	65.5	65.8
TextVQA_验证集	73.4	74.3	74.4	79.7	79.1
OCRBench	784	804	788	809	818
MME_总和	1876.8	2138.2	2059.8	1872.0	2109.8
RealWorldQA	57.3	60.1	60.7	62.6	63.1
AI2D_测试集	74.1	74.9	74.7	78.9	79.3
MMMU_验证集	36.3	43.6	47.9	41.1	43.1
MMVet_GPT-4-Turbo	39.5	60.8	51.0	49.5	53.8
HallBench_平均值	37.9	42.6	41.9	41.7	45.8
MathVista_测试小集	46.3	51.3	58.6	43.0	54.7
MMstar	50.1	53.7	54.3	48.0	56.4

📋 待办事项

[ ] 支持 vLLM 推理
[ ] 提供 AWQ 量化权重
[ ] 提供微调脚本

📄 许可证

代码根据 Apache 2.0 许可证发布。
预训练模型权重根据知识共享署名-非商业性使用 4.0 国际许可协议发布。
该服务仅供非商业研究预览使用，并受以下许可证和条款约束：
- Qwen2.5-1.5B-Instruct 模型许可证：Apache-2.0
- PaliGemma 模型许可证：Gemma 许可证