Ko Deplot

🚀 ko-deplot

ko-deplot is a Korean Visual-QA model based on Google's Pix2Struct architecture. It was fine-tuned from Deplot using Korean chart image-text pairs, aiming to provide accurate answers to visual questions in Korean.

🚀 Quick Start

You can run a prediction by querying an input image together with a question as follows:

from transformers import Pix2StructProcessor, Pix2StructForConditionalGeneration
from PIL import Image

processor = Pix2StructProcessor.from_pretrained('nuua/ko-deplot')
model = Pix2StructForConditionalGeneration.from_pretrained('nuua/ko-deplot')

IMAGE_PATH = "LOCAL_PATH_TO_IMAGE"
image = Image.open(IMAGE_PATH)

inputs = processor(images=image, text="Generate underlying data table of the figure below:", return_tensors="pt")
predictions = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(predictions[0], skip_special_tokens=True))

✨ Features

• Language Support: Specialized for Korean Visual Question Answering.
• Fine-tuned Model: Fine-tuned from Deplot with Korean chart image-text pairs.

📦 Installation

No specific installation steps are provided in the original document.

📚 Documentation

Model Information

Tokenizer Details

The model's tokenizer vocab was extended from 50,344 to 65,536 tokens using the following:

• Complete Korean Jamo
• Additional Korean Jamo
• Ko-Electra tokens

Training Details

Training Data

Synthetic chart data from three libraries were used:

• GenPlot
• Chart.js
• Plotly

Training Procedure

The model was first exposed to a short warmup stage, following its original paper. It was then trained using the chart data for 50,000 steps.

Technical Specifications

Hardware

ko-deplot was trained by using A100 80G.

📄 License

This project is licensed under the apache-2.0 license.

📞 Contact

Any questions and suggestions, please use the discussion tab. If you want to contact us directly, email robin@nuua.ai.