Phi 3 Small 8k Instruct Onnx Cuda

🚀 Phi-3 Small-8K-Instruct ONNX CUDA models

This repository hosts optimized versions of Phi-3-small-8k-instruct. It aims to accelerate inference with ONNX Runtime on machines equipped with NVIDIA GPUs.

Phi-3 Small is a 7B parameter, lightweight, state-of-the-art open model. It's trained with Phi-3 datasets, which consist of synthetic data and filtered publicly available website data, emphasizing high - quality and reasoning - dense properties. The model belongs to the Phi-3 family, with the small version having two variants: 8K and 128K, representing the context lengths (in tokens) they can support.

The base model has undergone a post - training process that includes supervised fine - tuning and direct preference optimization for instruction following and safety measures. When evaluated against benchmarks for common sense, language understanding, math, code, long context, and logical reasoning, Phi-3-Small-8K-Instruct demonstrated robust and state - of - the - art performance among models of the same size and next - size - up.

The optimized variants of the Phi-3 Small models are published in ONNX format and can run with ONNX Runtime on GPUs across various devices, including server platforms, Windows, and Linux.

🚀 Quick Start

To support the Phi-3 models across a range of devices, platforms, and EP backends, we introduce a new API to wrap several aspects of generative AI inferencing. This API makes it easy to integrate LLMs into your app. To run the early version of these models with ONNX, follow the steps here. You can also test the models with this chat app.

✨ Features

ONNX Models

Here are some of the optimized configurations we've added:

1. ONNX model for FP16 CUDA: Designed for NVIDIA GPUs.
2. ONNX model for INT4 CUDA: An ONNX model for NVIDIA GPUs using int4 quantization via RTN.

Note: If you're short on disk space, you can use the Hugging Face CLI to download sub - folders instead of all models. The FP16 model is recommended for larger batch sizes, while the INT4 model optimizes performance for lower batch sizes.

Example:

# Download just the FP16 model
$ huggingface-cli download microsoft/Phi-3-small-8k-instruct-onnx-cuda --include cuda-fp16/* --local-dir .  --local-dir-use-symlinks False

📚 Documentation

Hardware Supported

The ONNX models are tested on:

• 1 A100 GPU, SKU: Standard_ND96amsr_A100_v4 (CUDA)

Minimum Configuration Required:

• CUDA: NVIDIA GPU with Compute Capability >= 7.5

Model Description

Additional Details

• Phi-3 Small, Medium, and Vision Blog and Phi-3 Mini Blog
• Phi-3 Model Blog Link
• Phi-3 Model Card
• Phi-3 Technical Report
• Phi-3 on Azure AI Studio

Performance Metrics

Phi-3 Small-8K-Instruct performs better with ONNX Runtime compared to PyTorch for all batch size and prompt length combinations. For FP16 CUDA, ORT is up to 4X faster than PyTorch, while with INT4 CUDA, it's up to 10.9X faster than PyTorch.

The table below shows the average throughput of the first 256 tokens generated (tps) for FP16 and INT4 precisions on CUDA as measured on 1 A100 80GB GPU, SKU: Standard_ND96amsr_A100_v4.

Package Versions

📄 License

The model is released under the MIT license.

Appendix

Model Card Contact

parinitarahi, kvaishnavi, natke

Contributors

Kunal Vaishnavi, Sunghoon Choi, Yufeng Li, Tianlei Wu, Sheetal Arun Kadam, Rui Ren, Baiju Meswani, Natalie Kershaw, Parinita Rahi