Blip2zh Chatglm 6b

🚀 Model Card for blip2zh-chatglm-6b

This model is a Chinese multi-modal chat model based on BLIP2, offering basic image understanding capabilities. Since BLIP2's training method doesn't fine-tune the language model, its behavior in pure text conversations remains consistent with the original ChatGLM.

📚 Documentation

Model Details

The blip2zh-chatglm-6b is a Chinese multi-modal chat model trained on BLIP2. It has basic image understanding capabilities. Since the training method of BLIP2 does not fine-tune the language model, its behavior in pure text conversations can remain consistent with the original ChatGLM.

Note: Since the current model has only undergone two-stage image-text alignment pre-training with BLIP2 and does not include training for specific downstream tasks such as VQA or instruction fine-tuning, it is still prone to generating unexpected content.

• BLIP2 base model: bert-base-chinese
• Vision encoder: eva-clip-vit-g
• Language model: chatglm-6b at commit

Model Sources

• Training Code: BLIP2 training code, based on LAVIS
• webui: A web UI implemented with Gradio
• api: An API service implemented with FastAPI, which can be deployed locally and also supports some other types of locally deployable language models.

Uses

The model parameters include an image encoder, BLIP2, and chatglm-6b.

For loading the model and performing inference, you can refer to the implementation in api.

Some examples are also available.

Limitations

Limited by the Chinese dataset, the current image understanding ability is still limited and may generate irrelevant or incorrect content. Currently, multi-turn conversation training and instruction fine-tuning are not introduced. Multi-turn conversations may be interfered with by the context. Moreover, it is also limited by the conversation effect of ChatGLM-6B itself.

Training Details

Training Data

• laion-2b-chinese: We only selected 670k image-text pairs with high CLIP scores and sampled some data for training.
• coco-zh
• flickr8k-zh

Training Procedure

Based on the two-stage training method of BLIP2.

Demos

🔧 Technical Details

⚠️ Important Note

Since the current model has only undergone two-stage image-text alignment pre-training with BLIP2 and does not include training for specific downstream tasks such as VQA or instruction fine-tuning, it is still prone to generating unexpected content.

💡 Usage Tip

For loading the model and performing inference, you can refer to the implementation in api. Some examples are also available for reference.