Internvit 6B 448px V1 2

🚀 InternViT-6B-448px-V1-2

InternViT-6B-448px-V1-2是全新發布的權重版本。在InternVL 1.2更新中，涉及對InternViT-6B模型的持續預訓練。具體而言，將InternViT-6B的分辨率從224提升至448，並與[Nous-Hermes-2-Yi-34B]((https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B)進行集成。

[📂 GitHub] [📜 InternVL 1.0] [📜 InternVL 1.5] [📜 Mini-InternVL] [📜 InternVL 2.5]

[🆕 Blog] [🗨️ Chat Demo] [🤗 HF Demo] [🚀 Quick Start] [📖 Documents]

✨ 主要特性

• 分辨率提升：將InternViT-6B的分辨率從224提高到448，增強了模型處理高分辨率圖像的能力。
• 集成優化：與[Nous-Hermes-2-Yi-34B]((https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B)集成，提升了模型的性能。
• 數據增強：在通用圖像描述數據集的基礎上，加入額外的OCR數據，提升了模型的OCR能力。
• 參數優化：通過丟棄最後3個塊，將模型參數從59億減少到55億，節省了GPU內存。

📦 安裝指南

文檔未提及具體安裝步驟，暫不提供。

💻 使用示例

基礎用法

import torch
from PIL import Image
from transformers import AutoModel, CLIPImageProcessor

model = AutoModel.from_pretrained(
    'OpenGVLab/InternViT-6B-448px-V1-2',
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True).cuda().eval()

image = Image.open('./examples/image1.jpg').convert('RGB')

image_processor = CLIPImageProcessor.from_pretrained('OpenGVLab/InternViT-6B-448px-V1-2')

pixel_values = image_processor(images=image, return_tensors='pt').pixel_values
pixel_values = pixel_values.to(torch.bfloat16).cuda()

outputs = model(pixel_values)

📚 詳細文檔

模型詳情

注意事項

InternViT-6B原本有48個塊，經實驗發現，使用倒數第4個塊之後的輸出對多模態大語言模型（MLLM）效果最佳。為了方便使用和節省GPU內存，直接丟棄了最後3個塊。現在，模型僅剩下45個塊，參數數量從59億減少到55億。因此，如果要基於此模型構建MLLM，請使用最後一層的特徵。

⚠️ 重要提示

根據經驗，InternViT V2.5系列更適合用於構建多模態大語言模型，而非傳統的計算機視覺任務。

📄 許可證

本項目採用MIT許可證發佈。

🔖 引用

如果您在研究中發現本項目有用，請考慮引用以下文獻：

@article{chen2024expanding,
  title={Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling},
  author={Chen, Zhe and Wang, Weiyun and Cao, Yue and Liu, Yangzhou and Gao, Zhangwei and Cui, Erfei and Zhu, Jinguo and Ye, Shenglong and Tian, Hao and Liu, Zhaoyang and others},
  journal={arXiv preprint arXiv:2412.05271},
  year={2024}
}
@article{gao2024mini,
  title={Mini-internvl: A flexible-transfer pocket multimodal model with 5\% parameters and 90\% performance},
  author={Gao, Zhangwei and Chen, Zhe and Cui, Erfei and Ren, Yiming and Wang, Weiyun and Zhu, Jinguo and Tian, Hao and Ye, Shenglong and He, Junjun and Zhu, Xizhou and others},
  journal={arXiv preprint arXiv:2410.16261},
  year={2024}
}
@article{chen2024far,
  title={How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites},
  author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},
  journal={arXiv preprint arXiv:2404.16821},
  year={2024}
}
@inproceedings{chen2024internvl,
  title={Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks},
  author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and others},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={24185--24198},
  year={2024}
}