Florence 2 Large TableDetection

🚀 Florence-2-large-TableDetection

This model is a multimodal language model fine-tuned for table detection in images with textual prompts, aiming to automate table detection in various applications.

🚀 Quick Start

This model is a fine - tuned version of microsoft/Florence-2-large-ft on ucsahin/pubtables-detection-1500-samples dataset. It achieves a loss of 0.7601 on the evaluation set.

The microsoft/Florence-2-large-ft can detect various objects in zero - shot setting with the task prompt "<OD>". Check Florence-2-large sample inference for inference usage. However, the ft - base model can't detect tables in a given image.

The following Colab notebook shows how to finetune the model with custom data for object detection:

Florence2-Object Detection-Finetuning-HF-Trainer.ipynb

✨ Features

• This model is a multimodal language model fine - tuned for detecting tables in images given textual prompts. It uses a combination of image and text inputs to predict bounding boxes around tables in the provided images.
• Its main purpose is to automate the table detection process in images. It can be used in various applications like document processing, data extraction, and image analysis, where identifying tables in images is crucial.

📦 Installation

There is no specific installation steps provided in the original document.

💻 Usage Examples

Basic Usage

In Transformers, you can load the model and perform inference as follows: (Note that trust_remote_code=True is needed to run the model. It will only download the external custom codes from the original HuggingFaceM4/Florence-2-DocVQA.)

from transformers import AutoProcessor, AutoModelForCausalLM
import matplotlib.pyplot as plt
import matplotlib.patches as patches

model_id = "ucsahin/Florence-2-large-TableDetection"
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, device_map="cuda") # load the model on GPU
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

def run_example(task_prompt, image, max_new_tokens=128):
    prompt = task_prompt
    inputs = processor(text=prompt, images=image, return_tensors="pt")
    generated_ids = model.generate(
      input_ids=inputs["input_ids"].cuda(),
      pixel_values=inputs["pixel_values"].cuda(),
      max_new_tokens=max_new_tokens,
      early_stopping=False,
      do_sample=False,
      num_beams=3,
    )
    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
    parsed_answer = processor.post_process_generation(
        generated_text,
        task=task_prompt,
        image_size=(image.width, image.height)
    )
    return parsed_answer

def plot_bbox(image, data):
   # Create a figure and axes
    fig, ax = plt.subplots()
    # Display the image
    ax.imshow(image)
    # Plot each bounding box
    for bbox, label in zip(data['bboxes'], data['labels']):
        # Unpack the bounding box coordinates
        x1, y1, x2, y2 = bbox
        # Create a Rectangle patch
        rect = patches.Rectangle((x1, y1), x2-x1, y2-y1, linewidth=1, edgecolor='r', facecolor='none')
        # Add the rectangle to the Axes
        ax.add_patch(rect)
        # Annotate the label
        plt.text(x1, y1, label, color='white', fontsize=8, bbox=dict(facecolor='red', alpha=0.5))
    # Remove the axis ticks and labels
    ax.axis('off')
    # Show the plot
    plt.show()

########### Inference
from datasets import load_dataset

dataset = load_dataset("ucsahin/pubtables-detection-1500-samples")

example_id = 5
image = dataset["train"][example_id]["image"]

parsed_answer = run_example("<OD>", image=image)
plot_bbox(image, parsed_answer["<OD>"])

📚 Documentation

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e - 06
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e - 08
• lr_scheduler_type: linear
• num_epochs: 10

Training results

Framework versions

• Transformers 4.42.0.dev0
• Pytorch 2.3.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1

📄 License

This model is released under the MIT license.

📦 Information Table