Phi 3.5 MoE Instruct

🚀 Phi-3.5-MoE

Phi-3.5-MoE is a lightweight, state-of-the-art open model. It's built on high - quality data, supports multilingual use, and has a 128K context length. It's optimized for various scenarios, especially those with memory and latency constraints.

🚀 Quick Start

Phi-3.5-MoE-instruct is integrated in the official version of transformers starting from 4.46.0. You can verify the current transformers version with: pip list | grep transformers.

The model is also available in Azure AI Studio.

✨ Features

• Lightweight and High - Performance: Built upon high - quality datasets, it offers strong performance in memory and compute - constrained environments.
• Multilingual Support: Supports a wide range of languages, including Arabic, Chinese, English, etc.
• Long Context Length: With a 128K context length, it can handle long - document tasks effectively.
• Rigorous Optimization: Underwent supervised fine - tuning, proximal policy optimization, and direct preference optimization for precise instruction adherence and safety.

📦 Installation

Requirements

Examples of required packages:

flash_attn==2.5.8
torch==2.3.1
accelerate==0.31.0
transformers==4.46.0

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline 

torch.random.manual_seed(0) 

model = AutoModelForCausalLM.from_pretrained( 
    "microsoft/Phi-3.5-MoE-instruct",  
    device_map="cuda",  
    torch_dtype="auto",  
    trust_remote_code=False,  
) 

tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3.5-MoE-instruct") 

messages = [ 
    {"role": "system", "content": "You are a helpful AI assistant."}, 
    {"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"}, 
    {"role": "assistant", "content": "Sure! Here are some ways to eat bananas and dragonfruits together: 1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey. 2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey."}, 
    {"role": "user", "content": "What about solving an 2x + 3 = 7 equation?"}, 
] 

pipe = pipeline( 
    "text-generation", 
    model=model, 
    tokenizer=tokenizer, 
) 

generation_args = { 
    "max_new_tokens": 500, 
    "return_full_text": False, 
    "temperature": 0.0, 
    "do_sample": False, 
} 

output = pipe(messages, **generation_args) 
print(output[0]['generated_text'])

📚 Documentation

Intended Uses

Primary Use Cases

The model is intended for commercial and research use in multiple languages. It's suitable for general - purpose AI systems and applications in:

1. Memory/compute constrained environments
2. Latency bound scenarios
3. Strong reasoning (especially code, math and logic)

It's designed to accelerate research on language and multimodal models as a building block for generative AI features.

Use Case Considerations

Our models are not specifically designed or evaluated for all downstream purposes. Developers should consider common limitations of language models, evaluate and mitigate for accuracy, safety, and fairness before using in a specific downstream use case, especially for high - risk scenarios. They should also adhere to applicable laws and regulations.

Tokenizer

Phi-3.5-MoE-Instruct supports a vocabulary size of up to 32064 tokens. The tokenizer files already provide placeholder tokens for downstream fine - tuning, and they can be extended up to the model's vocabulary size.

Input Formats

Given the nature of the training data, the Phi-3.5-MoE-instruct model is best suited for prompts using the chat format as follows:

<|system|>
You are a helpful assistant.<|end|>
<|user|>
How to explain Internet for a medieval knight?<|end|>
<|assistant|>

🔧 Technical Details

Benchmarks

To understand the capabilities, we compare Phi-3.5-MoE with a set of models over a variety of benchmarks using our internal benchmark platform.

High - level overview on representative benchmarks

Different categories across 80 public benchmark datasets

Overall, Phi-3.5-MoE with only 6.6B active parameters achieves a similar level of language understanding and math as much larger models. It outperforms bigger models in reasoning capability and is only behind GPT-4o-mini. However, it has limitations for certain tasks due to its size.

Multilingual

The table below highlights the multilingual capability of Phi-3.5-MoE on multilingual MMLU, MEGA, and multilingual MMLU - pro datasets.

Long Context

Phi-3.5-MoE supports 128K context length, capable of long - document/meeting summarization, long - document QA, and multilingual context retrieval.

RULER: a retrieval - based benchmark for long context understanding

RepoQA: a benchmark for long context code understanding

Training

Model

Training Datasets

Our training data includes a wide variety of sources, totaling 4.9 trillion tokens (including 10% multilingual), and is a combination of:

1. Publicly available documents filtered rigorously for quality, selected high - quality educational data, and code.
2. Newly created synthetic, “textbook - like” data for teaching math, coding, common sense reasoning, general knowledge of the world.
3. High - quality chat - format supervised data covering various topics to reflect human preferences.

📄 License

The model is released under the MIT license.