Intjudge

🚀 IntJudge-8B

IntJudge-v1, the public beta version of our IntJudge model, is now available. IntJudge is a judge model designed to evaluate open - ended multimodal generation methods. It uses the Dev Set of OpenING for training. With a novel data pipeline, IntJudge reaches an 82.42% agreement rate with human judgments, outperforming GPT - based evaluators by 11.34%. The IntJudge - 8B is built on the Qwen2 - VL - 7B - Instruct.

🚀 Quick Start

✨ Features

• IntJudge is a judge model for evaluating open - ended multimodal generation methods.
• It is trained on the Dev Set of OpenING.
• Achieves an 82.42% agreement rate with human judgments, outperforming GPT - based evaluators by 11.34%.
• Built on the Qwen2 - VL - 7B - Instruct.

📦 Installation

We recommend building from source using the command pip install git+https://github.com/huggingface/transformers to avoid the KeyError. You also need to install qwen - vl - utils.

pip install qwen-vl-utils

The code for using IntJudge for pairwise evaluation on interleaved image - text generation can be found at https://github.com/LanceZPF/OpenING.

📄 License

This project is licensed under the Apache - 2.0 license.

📚 Documentation

Citation

If you find our work helpful, please cite us:

@misc{zhou2024GATE,
      title={GATE OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation}, 
      author={Pengfei Zhou, Xiaopeng Peng, Jiajun Song, Chuanhao Li, Zhaopan Xu, Yue Yang, Ziyao Guo, Hao Zhang, Yuqi Lin, Yefei He, Lirui Zhao, Shuo Liu, Tianhua Li, Yuxuan Xie, Xiaojun Chang, Yu Qiao, Wenqi Shao, and Kaipeng Zhang},
      year={2024},
      eprint={2411.18499},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}