Phi 3 Mini 128k Instruct Onnx

🚀 Phi-3 Mini-128K-Instruct ONNX models

This repository hosts optimized versions of Phi-3-mini-128k-instruct to accelerate inference with ONNX Runtime. Phi-3 Mini is a lightweight, state - of - the - art open model. It's built on datasets used for Phi - 2, focusing on high - quality, reasoning - dense data. The model belongs to the Phi - 3 family, with 4K and 128K context - length variants. It has undergone rigorous enhancements, including supervised fine - tuning and direct preference optimization.

🚀 Quick Start

To easily get started with Phi - 3, you can use our newly introduced ONNX Runtime Generate() API. See here for instructions on how to run it.

✨ Features

• Optimized for ONNX: The models are published in ONNX format to run with ONNX Runtime on CPU and GPU across various devices, including servers, desktops, and mobile CPUs.
• DirectML Support: DirectML support enables hardware acceleration on Windows devices across AMD, Intel, and NVIDIA GPUs.
• New API: A new API is introduced to wrap generative AI inferencing, making it easy to integrate LLMs into your app.

📦 Installation

Not explicitly provided in the original README, so this section is skipped.

💻 Usage Examples

Basic Usage

python model-qa.py -m /*{YourModelPath}*/onnx/cpu_and_mobile/phi-3-mini-4k-instruct-int4-cpu -k 40 -p 0.95 -t 0.8 -r 1.0

*Input:*  <|user|>Tell me a joke<|end|><|assistant|>

*Output:*  Why don't scientists trust atoms?
           Because they make up everything!

This joke plays on the double meaning of "make up." In science, atoms are the fundamental building blocks of matter, literally making up everything. However, in a colloquial sense, "to make up" can mean to fabricate or lie, hence the humor.

📚 Documentation

ONNX Models

Here are some of the optimized configurations:

1. ONNX model for int4 DML: For AMD, Intel, and NVIDIA GPUs on Windows, quantized to int4 using AWQ.
2. ONNX model for fp16 CUDA: For NVIDIA GPUs.
3. ONNX model for int4 CUDA: For NVIDIA GPUs using int4 quantization via RTN.
4. ONNX model for int4 CPU and Mobile: For CPU and Mobile, using int4 quantization via RTN. There are two versions: Acc = 1 for improved accuracy and Acc = 4 for improved performance. For mobile devices, we recommend using the model with acc - level - 4.

More updates on AMD, and additional optimizations on CPU and Mobile will be added with the official ORT 1.18 release in early May.

Hardware Supported

The models are tested on:

• GPU SKU: RTX 4090 (DirectML)
• GPU SKU: 1 A100 80GB GPU, SKU: Standard_ND96amsr_A100_v4 (CUDA)
• CPU SKU: Standard F64s v2 (64 vcpus, 128 GiB memory)
• Mobile SKU: Samsung Galaxy S21

Minimum Configuration Required:

• Windows: DirectX 12 - capable GPU and a minimum of 4GB of combined RAM
• CUDA: NVIDIA GPU with Compute Capability >= 7.0

Model Description

Additional Details

• ONNX Runtime Optimizations Blog Link
• Phi - 3 Model Blog Link
• Phi - 3 Model Card
• Phi - 3 Technical Report

How to Get Started with the Model

To run the Phi - 3 models across various devices and platforms with different execution provider backends, we introduce a new API. For running the early versions of these models with ONNX Runtime, follow the steps here.

Performance Metrics

Phi - 3 Mini - 128K - Instruct performs better in ONNX Runtime than PyTorch for all batch size, prompt length combinations. For FP16 CUDA, ORT performs up to 5X faster than PyTorch, while with INT4 CUDA it's up to 9X faster than PyTorch.

The table below shows the average throughput of the first 256 tokens generated (tps) for FP16 and INT4 precisions on CUDA as measured on 1 A100 80GB GPU, SKU: Standard_ND96amsr_A100_v4.

Note: PyTorch compile and Llama.cpp currently do not support the Phi - 3 Mini - 128K - Instruct model.

Package Versions

Appendix - Activation Aware Quantization

AWQ works by identifying the top 1% most salient weights that are most important for maintaining accuracy and quantizing the remaining 99% of weights. This leads to less accuracy loss from quantization compared to many other quantization techniques. For more on AWQ, see here.

Model Card Contact

parinitarahi, kvaishnavi, natke

Contributors

Kunal Vaishnavi, Sunghoon Choi, Yufeng Li, Akshay Sonawane, Sheetal Arun Kadam, Rui Ren, Edward Chen, Scott McKay, Ryan Hill, Emma Ning, Natalie Kershaw, Parinita Rahi, Patrice Vignola, Chai Chaoweeraprasit, Logan Iyer, Vicente Rivera, Jacques Van Rhyn

📄 License

This project is licensed under the MIT license.