Block Diagram Global Information

🚀 Block Diagram Global Information Extractor

This model is designed to extract the overall summary of block diagram images, supporting both English and Korean languages.

🚀 Quick Start

This model was introduced in the paper "Unveiling the Power of Integration: Block Diagram Summarization through Local-Global Fusion" accepted at ACL 2024. The full code is available in this BlockNet GitHub repository.

✨ Features

• Trained using a transformer encoder and decoder architecture based on the Donut configuration.
• Supports both English and Korean languages.
• Comprises a visual encoder module and a text decoder module, both based on the Transformer architecture.

📦 Installation

The installation steps are not provided in the original README. If you want to use this model, you may need to refer to the BlockNet repository for detailed installation instructions.

💻 Usage Examples

Basic Usage

import os
from PIL import Image
import torch
from donut import DonutModel

# Load the pre-trained model
model = DonutModel.from_pretrained("shreyanshu09/block_diagram_global_information") 

# Move the model to GPU if available
if torch.cuda.is_available():
    model.half()
    device = torch.device("cuda:0")
    model.to(device)
    
# Function to process a single image
def process_image(image_path):
    # Load and process the image
    image = Image.open(image_path)
    task_name = os.path.basename('/block_diagram_global_information/dataset/c2t_data/')                  # Create empty folder anywhere
    result = model.inference(image=image, prompt=f"<s_{task_name}>")["predictions"][0]

    # Extract the relevant information from the result
    if 'c2t' in result:
        return result['c2t']
    else:
        return result['text_sequence']

# Example usage
image_path = 'image.png'                  # Input image file
result = process_image(image_path)

📚 Documentation

Model description

This model is trained using a transformer encoder and decoder architecture, based on the configuration specified in Donut, to extract the overall summary of block diagram images. It supports both English and Korean languages. The straightforward architecture comprises a visual encoder module and a text decoder module, both based on the Transformer architecture.

Training dataset

📄 License

The content of this project itself is licensed under the [Attribution - NonCommercial - ShareAlike 4.0 International (CC BY - NC - SA 4.0)](https://creativecommons.org/licenses/by - nc - sa/4.0/).

Contact

If you have any questions about this work, please contact Shreyanshu Bhushan using the following email addresses: shreyanshubhushan@gmail.com.