Phi 3 Mini 4k Instruct Onnx

🚀 Phi-3 Mini-4K-Instruct ONNX models

This repository hosts optimized versions of Phi-3-mini-4k-instruct to accelerate inference with ONNX Runtime.

Phi-3 Mini is a lightweight, state-of-the-art open model. It's built upon the datasets used for Phi-2, which include synthetic data and filtered websites, with a focus on very high-quality, reasoning-dense data. This model belongs to the Phi-3 model family. The mini version comes in two variants: 4K and 128K, representing the context length (in tokens) it can support. The model has undergone a rigorous enhancement process, incorporating both supervised fine-tuning and direct preference optimization to ensure precise instruction adherence and robust safety measures.

The optimized Phi-3 Mini models are published here in ONNX format. They can run with ONNX Runtime on CPU and GPU across various devices, including server platforms, Windows, Linux and Mac desktops, and mobile CPUs, using the precision best suited to each target.

DirectML support enables developers to bring hardware acceleration to Windows devices at scale across AMD, Intel, and NVIDIA GPUs. Alongside DirectML, ONNX Runtime provides cross-platform support for Phi-3 Mini across a range of devices for CPU, GPU, and mobile use.

To easily start using Phi-3, you can utilize our newly introduced ONNX Runtime Generate() API. Refer to here for instructions on how to run it.

✨ Features

• Optimized versions of Phi-3 Mini-4K-Instruct for ONNX Runtime.
• Support for multiple hardware platforms including Windows, CUDA, CPU, and mobile.
• New ONNX Runtime Generate() API for easy integration.
• Different quantization configurations for better performance and accuracy trade - off.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

python model-qa.py -m /*{YourModelPath}*/onnx/cpu_and_mobile/phi-3-mini-4k-instruct-int4-cpu -k 40 -p 0.95 -t 0.8 -r 1.0

*Input:*  <|user|>Tell me a joke<|end|><|assistant|>

*Output:*  Why don't scientists trust atoms?
           Because they make up everything!

This joke plays on the double meaning of "make up." In science, atoms are the fundamental building blocks of matter, literally making up everything. However, in a colloquial sense, "to make up" can mean to fabricate or lie, hence the humor.

📚 Documentation

ONNX Models

Here are some of the optimized configurations we've added:

1. ONNX model for int4 DML: An ONNX model for AMD, Intel, and NVIDIA GPUs on Windows, quantized to int4 using AWQ.
2. ONNX model for fp16 CUDA: An ONNX model you can use to run on your NVIDIA GPUs.
3. ONNX model for int4 CUDA: An ONNX model for NVIDIA GPUs using int4 quantization via RTN.
4. ONNX model for int4 CPU and Mobile: An ONNX model for CPU and mobile using int4 quantization via RTN. Two versions are uploaded to balance latency vs. accuracy. Acc=1 is targeted at improved accuracy, while Acc=4 is for improved performance. For mobile devices, we recommend using the model with acc - level - 4.

More updates on AMD, and additional optimizations on CPU and Mobile will be added with the official ORT 1.18 release in early May. Stay tuned!

Hardware Supported

The models are tested on:

• GPU SKU: RTX 4090 (DirectML)
• GPU SKU: 1 A100 80GB GPU, SKU: Standard_ND96amsr_A100_v4 (CUDA)
• CPU SKU: Standard F64s v2 (64 vcpus, 128 GiB memory)
• Mobile SKU: Samsung Galaxy S21

Minimum Configuration Required:

• Windows: DirectX 12 - capable GPU and a minimum of 4GB of combined RAM
• CUDA: NVIDIA GPU with Compute Capability >= 7.0

Model Description

Additional Details

• ONNX Runtime Optimizations Blog Link
• Phi-3 Model Blog Link
• Phi-3 Model Card
• Phi-3 Technical Report

How to Get Started with the Model

To enable running the Phi-3 models across a range of devices and platforms with various execution provider backends, we've introduced a new API to wrap several aspects of generative AI inferencing. This API makes it easy to integrate LLMs into your app. For running the early version of these models with ONNX Runtime, follow the steps here.

🔧 Technical Details

Performance Metrics

DirectML on Windows

We measured the performance of DirectML and ONNX Runtime's new Generate() API with Phi-3 Mini quantized with Activation-Aware Quantization (AWQ) and a block size of 128 on Windows. Our test machine had an NVIDIA GeForce RTX 4090 GPU and an Intel Core i9 - 13900K CPU. DirectML allows developers not only to achieve great performance but also to deploy models across the entire Windows ecosystem with support from AMD, Intel, and NVIDIA. Best of all, AWQ means that developers can achieve this scale while maintaining high model accuracy.

Stay tuned for additional performance improvements in the coming weeks thanks to optimized drivers from our hardware partners, along with additional updates to the ONNX Runtime Generate() API.

CUDA

Phi-3 Mini-4K-Instruct performs better in ONNX Runtime than PyTorch for all batch size and prompt length combinations. For FP16 CUDA, ORT performs up to 5X faster than PyTorch, while with INT4 CUDA it's up to 10X faster than PyTorch. It is also up to 3X faster than Llama.cpp for large batch sizes.

The table below shows the average throughput of the first 256 tokens generated (tps) for FP16 and INT4 precisions on CUDA as measured on 1 A100 80GB GPU, SKU: Standard_ND96amsr_A100_v4.

CPU

The table below shows the average throughput of the first 256 tokens generated (tps) for INT4 precision on CPU as measured on a Standard F64s v2 (64 vcpus, 128 GiB memory).

Package Versions

Appendix

Activation Aware Quantization

AWQ works by identifying the top 1% most salient weights that are most important for maintaining accuracy and quantizing the remaining 99% of weights. This results in less accuracy loss from quantization compared to many other quantization techniques. For more on AWQ, see here.

📄 License

The model is released under the MIT license.

Model Card Contact

parinitarahi, kvaishnavi, natke

Contributors

Kunal Vaishnavi, Sunghoon Choi, Yufeng Li, Akshay Sonawane, Sheetal Arun Kadam, Rui Ren, Edward Chen, Scott McKay, Ryan Hill, Emma Ning, Natalie Kershaw, Parinita Rahi, Patrice Vignola, Chai Chaoweeraprasit, Logan Iyer, Vicente Rivera, Jacques Van Rhyn