AKI 4B Phi 3.5 Mini

🚀 AKI Model Card

AKI is a multimodal foundation model that unlocks causal attention in the LLM into modality - mutual attention (MMA), addressing vision - language misalignment without extra parameters and increased training time.

📚 Documentation

Model Details

Model Descriptions

• Vision Encoder: [google/siglip - so400m - patch14 - 384](https://huggingface.co/google/siglip - so400m - patch14 - 384)
• Vision - Language Connector: Perceiver Resampler
• Language Decoder (LLM): [microsoft/Phi - 3.5 - mini - instruct](https://huggingface.co/microsoft/Phi - 3.5 - mini - instruct)
• Pretraining Datasets: [Blip3 - kale](https://huggingface.co/datasets/Salesforce/blip3 - kale) and [Blip3 - OCR - 200m](https://huggingface.co/datasets/Salesforce/blip3 - ocr - 200m)
• SFT Datasets: VQAv2, GQA, VSR, OCRVQA, A - OKVQA, ScienceQA, RefCOCO, RefCOCOg, RefCOCO+, VisualGnome, LLaVA - 150k

Model Sources

• Repository: GitHub
• Paper: arXiv

💻 Usage Examples

Input Format

Given the nature of the training data, the AKI model is best suited for prompts using the chat format as follows:

<|system|>
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.<|end|>
<|user|>
<image>
Describe the scene of this image.
<|end|>
<|assistant|>

The image captures a beautiful autumn day in a park, with a pathway covered in a vibrant carpet of fallen leaves. The leaves are in various shades of red, orange, yellow, and brown, creating a warm and colorful atmosphere. The path is lined with trees displaying beautiful autumn foliage, adding to the picturesque setting. ...

Inference Example

Please refer to the notebook for the zero - shot inference. To build a local demo website, please refer to local_demo.py.

💡 Usage Tip

For the training scripts, please refer to the GitHub repo.

📊 Evaluation Results

Main Comparisons with the Same Configurations (Table 1)

AKI - 4B (Table 2)

⚖️ Ethical Considerations

⚠️ Important Note

This section is mainly taken from the [xgen - mm](https://huggingface.co/Salesforce/xgen - mm - phi3 - mini - base - r - v1/blob/main/README.md) models.

This release is for research purposes only in support of an academic paper. Our models, datasets, and code are not specifically designed or evaluated for all downstream purposes. We strongly recommend users evaluate and address potential concerns related to accuracy, safety, and fairness before deploying this model. We encourage users to consider the common limitations of AI, comply with applicable laws, and leverage best practices when selecting use cases, particularly for high - risk scenarios where errors or misuse could significantly impact people’s lives, rights, or safety.

📄 License

Our code and weights are released under the CC - BY - NC 4.0 license.

The copyrights of the pre - training and finetuning data remain with the original data owner.

📚 Citations

@misc{wywang2025AKI,
      title={Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs}, 
      author={Wei-Yao Wang and Zhao Wang and Helen Suzuki and Yoshiyuki Kobayashi},
      year={2025},
      eprint={2503.02597},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2503.02597}, 
}