Phi 4 Multimodal Instruct

🚀 Phi-4-multimodal-instruct

Phi-4-multimodal-instruct is a lightweight open multimodal foundation model. It processes text, image, and audio inputs, generating text outputs, and has a 128K token context length. It supports multiple languages and various multimodal tasks, enhancing research and application development in the field of AI.

🚀 Quick Start

The provided README does not contain specific quick - start content.

✨ Features

• Multimodal Processing: Capable of handling text, image, and audio inputs, and generating text outputs.
• Multilingual Support: Supports a wide range of languages including Arabic, Chinese, English, etc.
• Strong Reasoning: Demonstrates strong reasoning abilities, especially in math and logic.
• Function and Tool Calling: Supports function and tool calling.
• Rich Use Cases: Applicable to various scenarios such as speech recognition, translation, summarization, and image understanding.

📦 Installation

The provided README does not contain installation steps.

💻 Usage Examples

The provided README does not contain code examples.

📚 Documentation

Model Summary

Phi-4-multimodal-instruct is a lightweight open multimodal foundation model that leverages the language, vision, and speech research and datasets used for Phi - 3.5 and 4.0 models. The model processes text, image, and audio inputs, generating text outputs, and comes with 128K token context length. The model underwent an enhancement process, incorporating both supervised fine - tuning, direct preference optimization and RLHF (Reinforcement Learning from Human Feedback) to support precise instruction adherence and safety measures.

The languages that each modal supports are the following:

• Text: Arabic, Chinese, Czech, Danish, Dutch, English, Finnish, French, German, Hebrew, Hungarian, Italian, Japanese, Korean, Norwegian, Polish, Portuguese, Russian, Spanish, Swedish, Thai, Turkish, Ukrainian
• Vision: English
• Audio: English, Chinese, German, French, Italian, Japanese, Spanish, Portuguese

📰 Phi-4-multimodal Microsoft Blog
📖 Phi-4-multimodal Technical Report
🏡 Phi Portal
👩‍🍳 Phi Cookbook
🖥️ Try It on Azure, Nvidia Playgroud
📱Huggingface Spaces Thoughts Organizer, Stories Come Alive, Phine Speech Translator

Phi-4: [multimodal-instruct | onnx]; mini-instruct;

Watch as Phi-4 Multimodal analyzes spoken language to help plan a trip to Seattle, demonstrating its advanced audio processing and recommendation capabilities.

See how Phi-4 Multimodal tackles complex mathematical problems through visual inputs, demonstrating its ability to process and solve equations presented in images.

Explore how Phi-4 Mini functions as an intelligent agent, showcasing its reasoning and task execution abilities in complex scenarios.

Intended Uses

Primary Use Cases

The model is intended for broad multilingual and multimodal commercial and research use. The model provides uses for general purpose AI systems and applications which require:

1. Memory/compute constrained environments
2. Latency bound scenarios
3. Strong reasoning (especially math and logic)
4. Function and tool calling
5. General image understanding
6. Optical character recognition
7. Chart and table understanding
8. Multiple image comparison
9. Multi - image or video clip summarization
10. Speech recognition
11. Speech translation
12. Speech QA
13. Speech summarization
14. Audio understanding

The model is designed to accelerate research on language and multimodal models, for use as a building block for generative AI powered features.

Use Case Considerations

The model is not specifically designed or evaluated for all downstream purposes. Developers should consider common limitations of language models and multimodal models, as well as performance difference across languages, as they select use cases, and evaluate and mitigate for accuracy, safety, and fairness before using within a specific downstream use case, particularly for high - risk scenarios. Developers should be aware of and adhere to applicable laws or regulations (including but not limited to privacy, trade compliance laws, etc.) that are relevant to their use case.

Nothing contained in this Model Card should be interpreted as or deemed a restriction or modification to the license the model is released under.

Release Notes

This release of Phi-4-multimodal-instruct is based on valuable user feedback from the Phi - 3 series. Previously, users could use a speech recognition model to talk to the Mini and Vision models. To achieve this, users needed to use a pipeline of two models: one model to transcribe the audio to text, and another model for the language or vision tasks. This pipeline means that the core model was not provided the full breadth of input information – e.g. cannot directly observe multiple speakers, background noises, jointly align speech, vision, language information at the same time on the same representation space.

With Phi-4-multimodal-instruct, a single new open model has been trained across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network. The model employed new architecture, larger vocabulary for efficiency, multilingual, and multimodal support, and better post - training techniques were used for instruction following and function calling, as well as additional data leading to substantial gains on key multimodal capabilities.

It is anticipated that Phi-4-multimodal-instruct will greatly benefit app developers and various use cases. The enthusiastic support for the Phi - 4 series is greatly appreciated. Feedback on Phi - 4 is welcomed and crucial to the model's evolution and improvement. Thank you for being part of this journey!

Model Quality

To understand the capabilities, Phi-4-multimodal-instruct was compared with a set of models over a variety of benchmarks using an internal benchmark platform (See Appendix A for benchmark methodology). Users can refer to the Phi-4-Mini-Instruct model card for details of language benchmarks. At the high - level overview of the model quality on representative speech and vision benchmarks:

Speech

The Phi-4-multimodal-instruct was observed as:

• Having strong automatic speech recognition (ASR) and speech translation (ST) performance, surpassing expert ASR model WhisperV3 and ST models SeamlessM4T - v2 - Large.
• Ranking number 1 on the Huggingface OpenASR leaderboard with word error rate 6.14% in comparison with the current best model 6.5% as of Jan 17, 2025.
• Being the first open - sourced model that can perform speech summarization, and the performance is close to GPT4o.
• Having a gap with close models, e.g. Gemini - 1.5 - Flash and GPT - 4o - realtime - preview, on speech QA task. Work is being undertaken to improve this capability in the next iterations.

Speech Recognition (lower is better)

The performance of Phi-4-multimodal-instruct on the aggregated benchmark datasets:

The performance of Phi-4-multimodal-instruct on different languages, averaging the WERs of CommonVoice and FLEURS:

Speech Translation (higher is better)

Translating from German, Spanish, French, Italian, Japanese, Portugues, Chinese to English:

Translating from English to German, Spanish, French, Italian, Japanese, Portugues, Chinese. Noted that WhiperV3 does not support this capability:

Speech Summarization (higher is better)

Speech QA

MT bench scores are scaled by 10x to match the score range of MMMLU:

Audio Understanding

AIR bench scores are scaled by 10x to match the score range of MMAU:

Vision

Vision - Speech tasks

Phi-4-multimodal-instruct is capable of processing both image and audio together, the following table shows the model quality when the input query for vision content is synthetic speech on chart/table understanding and document reasoning tasks. Compared to other existing state - of - the - art omni models that can enable audio and visual signal as input, Phi-4-multimodal-instruct achieves much stronger performance on multiple benchmarks.

Vision tasks

To understand the vision capabilities, Phi-4-multimodal-instruct was compared with a set of models over a variety of zero - shot benchmarks using an internal benchmark platform. At the high - level overview of the model quality on representative benchmarks:

Visual Perception

Below are the comparison results on existing multi - image tasks. On average, Phi-4-multimodal-instruct outperforms competitor models of the same size and competitive with much bigger models on multi - frame capabilities. BLINK is an aggregated benchmark with 14 visual tasks that humans can solve very quickly but are still hard for current multimodal LLMs.

📄 License

This model is released under the MIT license. License Link