đ llava-llama-3-8b-v1_1
llava-llama-3-8b-v1_1 is a fine - tuned LLaVA model for image - to - text tasks, offering high - performance visual understanding capabilities.
đ Quick Start
Download models
wget https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-gguf/resolve/main/llava-llama-3-8b-v1_1-mmproj-f16.gguf
wget https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-gguf/resolve/main/llava-llama-3-8b-v1_1-f16.gguf
wget https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-gguf/resolve/main/llava-llama-3-8b-v1_1-int4.gguf
wget https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-gguf/resolve/main/OLLAMA_MODELFILE_F16
wget https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-gguf/resolve/main/OLLAMA_MODELFILE_INT4
Chat by ollama
ollama create llava-llama3-f16 -f ./OLLAMA_MODELFILE_F16
ollama run llava-llama3-f16 "xx.png Describe this image"
ollama create llava-llama3-int4 -f ./OLLAMA_MODELFILE_INT4
ollama run llava-llama3-int4 "xx.png Describe this image"
Chat by llama.cpp
- Build llama.cpp (docs).
- Build
./llava-cli
(docs).
Note: llava-llama-3-8b-v1_1 uses the Llama-3-instruct chat template.
./llava-cli -m ./llava-llama-3-8b-v1_1-f16.gguf --mmproj ./llava-llama-3-8b-v1_1-mmproj-f16.gguf --image YOUR_IMAGE.jpg -c 4096 -e -p "<|start_header_id|>user<|end_header_id|>\n\n<image>\nDescribe this image<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
./llava-cli -m ./llava-llama-3-8b-v1_1-int4.gguf --mmproj ./llava-llama-3-8b-v1_1-mmproj-f16.gguf --image YOUR_IMAGE.jpg -c 4096 -e -p "<|start_header_id|>user<|end_header_id|>\n\n<image>\nDescribe this image<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
Reproduce
Please refer to docs.
⨠Features
đ Documentation
Model
llava-llama-3-8b-v1_1 is a LLaVA model fine - tuned from meta-llama/Meta-Llama-3-8B-Instruct and CLIP-ViT-Large-patch14-336 with ShareGPT4V-PT and InternVL-SFT by XTuner.
Note: This model is in GGUF format.
Resources:
Details
Property |
Details |
Datasets |
Lin - Chen/ShareGPT4V |
Pipeline Tag |
image - to - text |
Model |
Visual Encoder |
Projector |
Resolution |
Pretraining Strategy |
Fine - tuning Strategy |
Pretrain Dataset |
Fine - tune Dataset |
LLaVA - v1.5 - 7B |
CLIP - L |
MLP |
336 |
Frozen LLM, Frozen ViT |
Full LLM, Frozen ViT |
LLaVA - PT (558K) |
LLaVA - Mix (665K) |
LLaVA - Llama - 3 - 8B |
CLIP - L |
MLP |
336 |
Frozen LLM, Frozen ViT |
Full LLM, LoRA ViT |
LLaVA - PT (558K) |
LLaVA - Mix (665K) |
LLaVA - Llama - 3 - 8B - v1.1 |
CLIP - L |
MLP |
336 |
Frozen LLM, Frozen ViT |
Full LLM, LoRA ViT |
ShareGPT4V - PT (1246K) |
InternVL - SFT (1268K) |
Results
Model |
MMBench Test (EN) |
MMBench Test (CN) |
CCBench Dev |
MMMU Val |
SEED - IMG |
AI2D Test |
ScienceQA Test |
HallusionBench aAcc |
POPE |
GQA |
TextVQA |
MME |
MMStar |
LLaVA - v1.5 - 7B |
66.5 |
59.0 |
27.5 |
35.3 |
60.5 |
54.8 |
70.4 |
44.9 |
85.9 |
62.0 |
58.2 |
1511/348 |
30.3 |
LLaVA - Llama - 3 - 8B |
68.9 |
61.6 |
30.4 |
36.8 |
69.8 |
60.9 |
73.3 |
47.3 |
87.2 |
63.5 |
58.0 |
1506/295 |
38.2 |
LLaVA - Llama - 3 - 8B - v1.1 |
72.3 |
66.4 |
31.6 |
36.8 |
70.1 |
70.0 |
72.9 |
47.7 |
86.4 |
62.6 |
59.0 |
1469/349 |
45.1 |
đ License
Citation
@misc{2023xtuner,
title={XTuner: A Toolkit for Efficiently Fine-tuning LLM},
author={XTuner Contributors},
howpublished = {\url{https://github.com/InternLM/xtuner}},
year={2023}
}