đ Cerule - A Tiny Mighty Vision Model
Cerule is a compact yet powerful vision language model built upon Google's Gemma - 2b and SigLIP, offering remarkable performance in image - text - to - text tasks.
ââââââââââââââââââââââ âââ ââââââ ââââââââ
âââââââââââââââââââââââââââ ââââââ ââââââââ
âââ ââââââ âââââââââââ ââââââ ââââââ
âââ ââââââ âââââââââââ ââââââ ââââââ
âââââââââââââââââââ ââââââââââââââââââââââââââââ
ââââââââââââââââââ âââ âââââââ ââââââââââââââââ
We trained and released "Cerule", a small but powerful Vision Language Model based on Google's newly released Gemma - 2b and Google's SigLIP.
- Pretraining stage: 650K images (A LAION Subset)
- Finetuning stage: 695K images (SVIT - mix - 665K - Bunny mix modified by BAAI)
The training setup was 4xA100's 80GB, taking approximately 6 hours for pretraining and about 13 hours for finetuning. We modified and adapted the training code from Bunny.
⨠Features
- Powerful Base: Built on Google's Gemma - 2b and SigLIP, leveraging their advanced capabilities.
- Efficient Training: Trained on specific image subsets with a reasonable time cost on
4xA100's 80GB setup.
- Versatile Inference: Allows for CLI - based inference with provided repositories.
đĻ Installation
The README does not provide specific installation steps, so this section is skipped.
đģ Usage Examples
Image Description Examples
| Image |
Example |
 |
Describe the image
The image is a playful and surreal depiction of a man in a space suit, sitting on a chair and holding a green beer bottle. The man is wearing a white space suit, complete with a helmet and gloves. His feet are clad in black and white shoes, and he is placed on a sandy surface. The background features a large, blue planet, with a moon and a star visible in the sky. |
 |
Who are the characters in the image?
The image features three characters, two of them are Mario and Luigi, and the third one is Yoshi.
Describe the actions of the characters
The Mario and Luigi characters are holding their arms out, as if they are waving. Yoshi is standing on its own, with its arms folded. |
 |
What's funny about this image?
The image is quite humorous as it depicts a man ironing clothes on the back of a yellow taxi cab. This is not a typical sight you'd expect to see in everyday life. |
đ Documentation
Training
Training code Released !!!
https://github.com/tensoic/Cerule
Inference
Clone the following repo and follow the instructions for a CLI - based inference.
https://github.com/tensoic/Cerule
đ§ Technical Details
- Training Data:
- Pretraining: 650K images (A LAION Subset)
- Finetuning: 695K images (SVIT - mix - 665K - Bunny mix modified by BAAI)
- Training Setup:
4xA100's 80GB, ~6 hours for pretraining and ~13 hours for finetuning
- Code Adaptation: Modified and adapted training code from Bunny
đ License
Model subject to Gemma (base model license) terms of use along with the underlying datasets (LAOIN and SVIT) subject to their respective licenses. All codes are Apache 2.0
đ Acknowledgments
We sincerely thank the amazing teams at Google, LLaVA, and BAAI without which this project would not have been possible!
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
