đ Llama-3.2V-11B-cot Model
Llama-3.2V-11B-cot is the first version of LLaVA-CoT, a visual language model capable of spontaneous, systematic reasoning.
đ Quick Start
You can use the inference code for Llama-3.2-11B-Vision-Instruct.
⨠Features
Llama-3.2V-11B-cot is a visual language model that can perform spontaneous and systematic reasoning. It is the first version of LLaVA-CoT.
đĻ Installation
No installation steps are provided in the original document, so this section is skipped.
đģ Usage Examples
No code examples are provided in the original document, so this section is skipped.
đ Documentation
Model Details
- License: apache-2.0
- Finetuned from model: meta-llama/Llama-3.2-11B-Vision-Instruct
Benchmark Results
| MMStar |
MMBench |
MMVet |
MathVista |
AI2D |
Hallusion |
Average |
| 57.6 |
75.0 |
60.3 |
54.8 |
85.7 |
47.8 |
63.5 |
Reproduction
To reproduce our results, you should use VLMEvalKit and the following settings.
| Parameter |
Value |
| do_sample |
True |
| temperature |
0.6 |
| top_p |
0.9 |
| max_new_tokens |
2048 |
You may change them in this file, line 80 - 83, and modify the max_new_tokens throughout the file.
â ī¸ Important Note
We follow the same settings as Llama-3.2-11B-Vision-Instruct, except that we extend the max_new_tokens to 2048. After you get the results, you should filter the model output and only keep the outputs between <CONCLUSION> and </CONCLUSION>. This shouldn't have any difference in theory, but empirically we observe some performance difference because the jugder GPT-4o can be inaccurate sometimes. By keeping the outputs between <CONCLUSION> and </CONCLUSION>, most answers can be direclty extracted using VLMEvalKit system, which can be much less biased.
Training Details
Training Data
The model is trained on the LLaVA-CoT-100k dataset.
Training Procedure
The model is finetuned on llama-recipes with the following settings. Using the same setting should accurately reproduce our results.
| Parameter |
Value |
| FSDP |
enabled |
| lr |
1e-5 |
| num_epochs |
3 |
| batch_size_training |
4 |
| use_fast_kernels |
True |
| run_validation |
False |
| batching_strategy |
padding |
| context_length |
4096 |
| gradient_accumulation_steps |
1 |
| gradient_clipping |
False |
| gradient_clipping_threshold |
1.0 |
| weight_decay |
0.0 |
| gamma |
0.85 |
| seed |
42 |
| use_fp16 |
False |
| mixed_precision |
True |
Bias, Risks, and Limitations
The model may generate biased or offensive content, similar to other VLMs, due to limitations in the training data. Technically, the model's performance in aspects like instruction following still falls short of leading industry models.
đ§ Technical Details
No specific technical implementation details (more than 50 words) are provided in the original document, so this section is skipped.
đ License
The model is licensed under the apache-2.0 license.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
