mPLUG-Owl: Open-Source Multilingual Visual-Language Model - Supports Image Understanding and Multi-Round Conversations

Mplug Owl Bloomz 7b Multilingual

Developed by MAGAer13

mPLUG-Owl is a multilingual vision-language model that supports image understanding and multi-turn dialogue, developed based on the BLOOMZ-7B architecture.

Image-to-Text

Transformers

Supports Multiple LanguagesOpen Source License:Apache-2.0 #Multimodal Dialogue #Multilingual Understanding #Image Caption Generation

Downloads 29

Release Time : 5/30/2023

Model Overview

This is a multimodal model capable of processing both image and text inputs, performing image-to-text generation tasks, and supporting multilingual interaction.

Model Features

Multilingual Support

Supports multilingual interaction in Chinese, English, French, Japanese, etc., with cross-language understanding capabilities.

Multimodal Understanding

Can process both image and text inputs simultaneously, understanding visual content and generating relevant descriptions.

Conversational Interaction

Supports multi-turn human-machine dialogue, enabling in-depth discussions based on image content.

Model Capabilities

Image Caption Generation

Multilingual Text Generation

Visual Question Answering

Meme Understanding

Multi-turn Dialogue

Use Cases

Content Understanding

Meme Interpretation

Explains the humor and cultural context of internet memes

Can generate meme analyses that align with human understanding

Creative Assistance

Image Annotation

Generates multilingual descriptive texts for images

Supports accurate image descriptions in multiple languages

🚀 MplugOwl Image-to-Text Model

MplugOwl is a powerful image-to-text model supporting multiple languages, enabling seamless conversion from images to text.

🚀 Quick Start

📦 Installation

Get the latest codebase from Github:

git clone https://github.com/X-PLUG/mPLUG-Owl.git

💻 Usage Examples

🔍 Basic Usage - Model Initialization

from transformers import AutoTokenizer
from mplug_owl.modeling_mplug_owl import MplugOwlForConditionalGeneration
from mplug_owl.processing_mplug_owl import MplugOwlImageProcessor, MplugOwlProcessor

pretrained_ckpt = 'MAGAer13/mplug-owl-bloomz-7b-multilingual'
model = MplugOwlForConditionalGeneration.from_pretrained(
    pretrained_ckpt,
    torch_dtype=torch.bfloat16,
)
image_processor = MplugOwlImageProcessor.from_pretrained(pretrained_ckpt)
tokenizer = AutoTokenizer.from_pretrained(pretrained_ckpt)
processor = MplugOwlProcessor(image_processor, tokenizer)

🔍 Advanced Usage - Model Inference

Prepare model inputs

# We use a human/AI template to organize the context as a multi-turn conversation.
# <image> denotes an image placeholder.
prompts = [
'''The following is a conversation between a curious human and AI assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
Human: <image>
Human: Explain why this meme is funny.
AI: ''']

# The image paths should be placed in the image_list and kept in the same order as in the prompts.
# We support urls, local file paths, and base64 string. You can customise the pre-processing of images by modifying the mplug_owl.modeling_mplug_owl.ImageProcessor
image_list = ['https://xxx.com/image.jpg']

Get response

# generate kwargs (the same in transformers) can be passed in the do_generate()
generate_kwargs = {
    'do_sample': True,
    'top_k': 5,
    'max_length': 512
}
from PIL import Image
images = [Image.open(_) for _ in image_list]
inputs = processor(text=prompts, images=images, return_tensors='pt')
inputs = {k: v.bfloat16() if v.dtype == torch.float else v for k, v in inputs.items()}
inputs = {k: v.to(model.device) for k, v in inputs.items()}
with torch.no_grad():
    res = model.generate(**inputs, **generate_kwargs)
sentence = tokenizer.decode(res.tolist()[0], skip_special_tokens=True)
print(sentence)

📄 License

This project is licensed under the Apache-2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご