Chartve

🚀 ChartVE (Chart Visual Entailment)

ChartVE is a visual entailment model designed to evaluate the factuality of a generated caption sentence against an input chart. It takes a chart figure and a caption sentence, then outputs an entailment probability.

🚀 Quick Start

ChartVE is a visual entailment model introduced in the paper "Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning". Its purpose is to evaluate the factuality of a generated caption sentence with respect to the input chart. The model takes a chart figure and a caption sentence as input and outputs an entailment probability. To compute the entailment probability, refer to the "💻 Usage Examples" section below. The underlying architecture of this model is UniChart.

Note that this model expects a caption sentence as textual input. For captions longer than one sentence, split the caption into multiple sentences, feed individual sentences to ChartVE, and then aggregate the scores.

💻 Usage Examples

Basic Usage

from transformers import DonutProcessor, VisionEncoderDecoderModel
from PIL import Image

model_name = "khhuang/chartve"
model = VisionEncoderDecoderModel.from_pretrained(model_name).cuda()
processor = DonutProcessor.from_pretrained(model_name)

image_path = "PATH_TO_IMAGE"

def format_query(sentence):
    return f"Does the image entails this statement: \"{sentence}\"?"

# Format text inputs
CAPTION_SENTENCE = "The state that has the highest number of population is California."
query = format_query(CAPTION_SENTENCE)

# Encode chart figure and tokenize text
img = Image.open(IMAGE_PATH)
pixel_values = processor(img.convert("RGB"), random_padding=False, return_tensors="pt").pixel_values
pixel_values = pixel_values.cuda()
decoder_input_ids = processor.tokenizer(query, add_special_tokens=False, return_tensors="pt", max_length=510).input_ids.cuda()#.squeeze(0)


outputs = model(pixel_values, decoder_input_ids=decoder_input_ids)

# positive_logit = outputs['logits'].squeeze()[-1,49922]
# negative_logit = outputs['logits'].squeeze()[-1,2334] 

# Probe the probability of generating "yes"
binary_entail_prob_positive = torch.nn.functional.softmax(outputs['logits'].squeeze()[-1,[2334, 49922]])[1].item()

# binary_entail_prob_positive corresponds to the computed probability that the chart entails the caption sentence.

📄 License

This project is licensed under the Apache-2.0 license.

📚 Documentation

Citation

@misc{huang-etal-2023-do,
    title = "Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning",
    author = "Huang, Kung-Hsiang  and
      Zhou, Mingyang and
      Chan, Hou Pong  and
      Fung, Yi R. and
      Wang, Zhenhailong and
      Zhang, Lingyu and
      Chang, Shih-Fu and
      Ji, Heng",
    year={2023},
    eprint={2312.10160},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}