Phi 4 Multimodal Instruct Onnx

🚀 Phi-4 Multimodal Instruct ONNX models

This is an ONNX version of the Phi-4 multimodal model, quantized to int4 precision for accelerated inference with ONNX Runtime.

🚀 Quick Start

✨ Features

• Tags: nlp, code, audio, automatic-speech-recognition, speech-summarization, speech-translation, visual-question-answering, phi-4-multimodal, phi, phi-4-mini

📦 Installation

For CPU

Stay tuned or follow this tutorial to generate your own ONNX models for CPU!

For CUDA

# Download the model directly using the Hugging Face CLI
huggingface-cli download microsoft/Phi-4-multimodal-instruct-onnx --include gpu/* --local-dir .

# Install the CUDA package of ONNX Runtime GenAI
pip install --pre onnxruntime-genai-cuda

# Please adjust the model directory (-m) accordingly 
curl https://raw.githubusercontent.com/microsoft/onnxruntime-genai/main/examples/python/phi4-mm.py -o phi4-mm.py
python phi4-mm.py -m gpu/gpu-int4-rtn-block-32 -e cuda

For DirectML

# Download the model directly using the Hugging Face CLI
huggingface-cli download microsoft/Phi-4-multimodal-instruct-onnx --include gpu/* --local-dir .

# Install the DML package of ONNX Runtime GenAI
pip install --pre onnxruntime-genai-directml

# Please adjust the model directory (-m) accordingly 
curl https://raw.githubusercontent.com/microsoft/onnxruntime-genai/main/examples/python/phi4-mm.py -o phi4-mm.py
python phi4-mm.py -m gpu/gpu-int4-rtn-block-32 -e dml

You will be prompted to provide any images, audios, and a prompt.

📚 Documentation

Model Description

Disclaimer: Model is only an optimization of the base model, any risk associated with the model is the responsibility of the user of the model. Please verify and test for you scenarios. There may be a slight difference in output from the base model with the optimizations applied.

Base Model

Phi-4-multimodal-instruct is a lightweight open multimodal foundation model that leverages the language, vision, and speech research and datasets used for Phi-3.5 and 4.0 models. The model processes text, image, and audio inputs, generating text outputs, and comes with 128K token context length. The model underwent an enhancement process, incorporating both supervised fine-tuning, and direct preference optimization to support precise instruction adherence and safety measures.

See details here

The performance of the text component is similar to the Phi-4 mini ONNX models

📄 License

This project is licensed under the MIT license.