Internvl2 5 1B MNN

🚀 InternVL2_5-1B-MNN

This model is a 4-bit quantized version of the MNN model exported from InternVL2_5-1B. It offers efficient performance and is suitable for text - generation tasks, especially in chat scenarios.

🚀 Quick Start

✨ Features

This model is a 4-bit quantized version of the MNN model exported from InternVL2_5-1B using llmexport.

📦 Installation

Download the model

# install huggingface
pip install huggingface

# shell download
huggingface download --model 'taobao-mnn/InternVL2_5-1B-MNN' --local_dir 'path/to/dir'

# SDK download
from huggingface_hub import snapshot_download
model_dir = snapshot_download('taobao-mnn/InternVL2_5-1B-MNN')

# git clone
git clone https://www.modelscope.cn/taobao-mnn/InternVL2_5-1B-MNN

💻 Usage Examples

Basic Usage

# clone MNN source
git clone https://github.com/alibaba/MNN.git

# compile
cd MNN
mkdir build && cd build
cmake .. -DMNN_LOW_MEMORY=true -DMNN_CPU_WEIGHT_DEQUANT_GEMM=true -DMNN_BUILD_LLM=true -DMNN_SUPPORT_TRANSFORMER_FUSE=true
make -j

# run
./llm_demo /path/to/InternVL2_5-1B-MNN/config.json prompt.txt

📚 Documentation

For more detailed information, please refer to MNN-LLM.

📄 License

This project is licensed under the Apache-2.0 license.