Bytedance BAGEL 7B MoT INT8

🚀 BAGEL - Unified Model for Multimodal Understanding and Generation

BAGEL is an open - source multimodal foundation model. It has 7B active parameters (14B total) and is trained on large - scale interleaved multimodal data. It outperforms many current top - tier open - source VLMs in multimodal understanding and generation tasks, and also shows excellent performance in image editing.

🚀 Quick Start

This repository hosts the model weights for BAGEL. For installation, usage instructions, and further documentation, please visit our GitHub repository.

✨ Features

INT8 Quantization

INT8 quant of ByteDance-Seed/BAGEL-7B-MoT

Model Architecture

BAGEL adopts a Mixture-of-Transformer-Experts (MoT) architecture to maximize the model's capacity to learn from richly diverse multimodal information. It utilizes two separate encoders to capture pixel - level and semantic - level features of an image. The overall framework follows a Next Group of Token Prediction paradigm, where the model is trained to predict the next group of language or visual tokens as a compression target.

Training Strategy

BAGEL scales MoT's capacity through Pre - training, Continued Training, and Supervised Finetuning on trillions of interleaved multimodal tokens spanning language, image, video, and web data. It surpasses open models on standard understanding and generation benchmarks and demonstrates advanced in - context multimodal abilities like free - form image editing, future frame prediction, 3D manipulation, world navigation, and sequential reasoning.

Emerging Properties

As we scale up BAGEL's pretraining with more multimodal tokens, we observe consistent performance gains across understanding, generation, and editing tasks. Different capabilities emerge at distinct training stages—multimodal understanding and generation appear early, followed by basic editing, while complex, intelligent editing emerges later. This staged progression suggests an emergent pattern, where advanced multimodal reasoning builds on well - formed foundational skills. Ablation studies further show that combining VAE and ViT features significantly improves intelligent editing, underscoring the importance of visual - semantic context in enabling complex multimodal reasoning and further supporting its role in the emergence of advanced capabilities.

📚 Documentation

Benchmarks

1. Visual Understanding

2. Text - to - Image Generation (GenEval)

3. Image Editing

📄 License

BAGEL is licensed under the Apache 2.0 license. It is finetuned from [Qwen2.5 - 7B - Instruct](https://huggingface.co/Qwen/Qwen2.5 - 7B - Instruct) and [siglip - so400m - 14 - 384 - flash - attn2](https://huggingface.co/HuggingFaceM4/siglip - so400m - 14 - 384 - flash - attn2) model, and uses the [FLUX.1 - schnell VAE model](https://huggingface.co/black - forest - labs/FLUX.1 - schnell), all under Apache 2.0.

📚 Citation

@article{deng2025bagel,
  title   = {Emerging Properties in Unified Multimodal Pretraining},
  author  = {Deng, Chaorui and Zhu, Deyao and Li, Kunchang and Gou, Chenhui and Li, Feng and Wang, Zeyu and Zhong, Shu and Yu, Weihao and Nie, Xiaonan and Song, Ziang and Shi, Guang and Fan, Haoqi},
  journal = {arXiv preprint arXiv:2505.14683},
  year    = {2025}
}

Information Table