Monkey

🚀 Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models

Monkey presents a training - efficient approach to effectively enhance the input resolution capacity up to 896 x 1344 pixels without starting from scratch in pretraining. To bridge the gap between simple text labels and high input resolution, a multi - level description generation method is proposed, which automatically offers rich information to guide the model in learning the contextual association between scenes and objects. With the synergy of these two designs, the model achieved excellent results on multiple benchmarks. By comparing with various LMMs, including GPT4V, Monkey demonstrates promising performance in image captioning by focusing on textual information and capturing fine details within images; its improved input resolution also enables remarkable performance in document images with dense text.

🚀 Quick Start

For a quick start, you can refer to the following sections to understand the features, set up the environment, run demos, evaluate, train, and perform inferences with the Monkey model.

✨ Features

• Contextual associations. Our method shows a superior ability to infer the relationships between targets more effectively when answering questions, leading to more comprehensive and insightful results.
• Support resolution up to 1344 x 896. Surpassing the standard 448 x 448 resolution commonly used for LMMs, this significant increase in resolution enhances the ability to discern and understand unnoticeable or tightly clustered objects and dense text.
• Enhanced general performance. Testing across 16 diverse datasets shows that the Monkey model performs impressively in tasks such as Image Captioning, General Visual Question Answering, Text - centric Visual Question Answering, and Document - oriented Visual Question Answering.

📦 Installation

conda create -n monkey python=3.9
conda activate monkey
git clone https://github.com/Yuliang - Liu/Monkey.git
cd ./Monkey
pip install -r requirements.txt

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "echo840/Monkey"
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map='cuda', trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
tokenizer.padding_side = 'left'
tokenizer.pad_token_id = tokenizer.eod_id
img_path = ""
question = ""
query = f'<img>{img_path}</img> {question} Answer: ' #VQA
# query = f'<img>{img_path}</img> Generate the detailed caption in English: ' #detailed caption

input_ids = tokenizer(query, return_tensors='pt', padding='longest')
attention_mask = input_ids.attention_mask
input_ids = input_ids.input_ids

pred = model.generate(
            input_ids=input_ids.cuda(),
            attention_mask=attention_mask.cuda(),
            do_sample=False,
            num_beams=1,
            max_new_tokens=512,
            min_new_tokens=1,
            length_penalty=1,
            num_return_sequences=1,
            output_hidden_states=True,
            use_cache=True,
            pad_token_id=tokenizer.eod_id,
            eos_token_id=tokenizer.eod_id,
            )
response = tokenizer.decode(pred[0][input_ids.size(1):].cpu(), skip_special_tokens=True).strip()
print(response)

Advanced Usage

Before 14/11/2023, we observed that for some random pictures, Monkey can achieve more accurate results than GPT4V.

We also provide the source code and the model weight for the original demo, allowing you to customize certain parameters for a more unique experience. The specific operations are as follows:

1. Make sure you have configured the environment.
2. You can choose to use the demo offline or online:

• Offline:
  • Download the Model Weight.
  • Modify DEFAULT_CKPT_PATH="pathto/Monkey" in the demo.py file to your model weight path.
  • Run the demo using the following command:

  python demo.py
  

• Online:
  • Run the demo and download model weights online with the following command:

  python demo.py -c echo840/Monkey 
  

📚 Documentation

Dataset

We have open - sourced the data generated by the multi - level description generation method. You can download it at Detailed Caption.

Evaluate

We offer evaluation code for 14 Visual Question Answering (VQA) datasets in the evaluate_vqa.py file, facilitating a quick verification of results. The specific operations are as follows:

1. Make sure you have configured the environment.
2. Modify sys.path.append("pathto/Monkey") to your model weight path.
3. Prepare the datasets required for evaluation.
4. Run the evaluation code.

Take ESTVQA as an example:

• Prepare data according to the following directory structure:

├── data
|   ├── estvqa
|       ├── test_image
|           ├── {image_path0}
|           ├── {image_path1}
|                 ·
|                 ·
|   ├── estvqa.jsonl

• Example of the format of each line of the annotated .jsonl file:

{"image": "data/estvqa/test_image/011364.jpg", "question": "What is this store?", "answer": "pizzeria", "question_id": 0}

• Modify the dictionary ds_collections:

ds_collections = {
    'estvqa_test': {
        'test': 'data/estvqa/estvqa.jsonl',
      'metric': 'anls',
        'max_new_tokens': 100,
    },
    ...
}

• Run the following command:

bash eval/eval.sh 'EVAL_PTH' 'SAVE_NAME'

Train

We also offer Monkey's model definition and training code, which you can explore above. You can execute the training code through executing finetune_ds_debug.sh.

⚠️ Important Note

Specify the path to your training data, which should be a json file consisting of a list of conversations.

Citing Monkey

If you wish to refer to the baseline results published here, please use the following BibTeX entries:

@article{li2023monkey,
  title={Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models},
  author={Li, Zhang and Yang, Biao and Liu, Qiang and Ma, Zhiyin and Zhang, Shuo and Yang, Jingxu and Sun, Yabo and Liu, Yuliang and Bai, Xiang},
  journal={arXiv preprint arXiv:2311.06607},
  year={2023}
}

Acknowledgement

[Qwen - VL](https://github.com/QwenLM/Qwen - VL.git): the codebase we built upon. Thanks for the authors of Qwen for providing the framework.

Copyright

We welcome suggestions to help us improve the Monkey. For any query, please contact Dr. Yuliang Liu: ylliu@hust.edu.cn. If you find something interesting, please also feel free to share with us through email or open an issue. Thanks!

Paper   |   Detailed Caption   |   Model Weight   | Model Weight in wisemodel