Detr Layout Detection

🚀 DETR-layout-detection

This model, cmarkea/detr-layout-detection, enables the extraction of various layouts (Text, Picture, Caption, Footnote, etc.) from a document image. It is a fine - tuned version of the [detr - resnet - 50](https://huggingface.co/facebook/detr - resnet - 50) model on the DocLayNet dataset. This model can jointly predict masks and bounding boxes for documentary objects, making it ideal for processing documentary corpora to be fed into an ODQA system.

It can extract 11 entities, namely: Caption, Footnote, Formula, List - item, Page - footer, Page - header, Picture, Section - header, Table, Text, and Title.

✨ Features

• Extract different layouts from document images.
• Jointly predict masks and bounding boxes for documentary objects.
• Ideal for ODQA system data pre - processing.

📦 Installation

Since the model is based on the transformers library, you can install it using the following command:

pip install transformers

💻 Usage Examples

Basic Usage

from transformers import AutoImageProcessor
from transformers.models.detr import DetrForSegmentation

img_proc = AutoImageProcessor.from_pretrained(
    "cmarkea/detr-layout-detection"
)
model = DetrForSegmentation.from_pretrained(
    "cmarkea/detr-layout-detection"
)

img: PIL.Image

with torch.inference_mode():
    input_ids = img_proc(img, return_tensors='pt')
    output = model(**input_ids)

threshold=0.4

segmentation_mask = img_proc.post_process_segmentation(
    output,
    threshold=threshold,
    target_sizes=[img.size[::-1]]
)

bbox_pred = img_proc.post_process_object_detection(
    output,
    threshold=threshold,
    target_sizes=[img.size[::-1]]
)

Example

📚 Documentation

Performance

In this section, we assess the model's performance by separately considering semantic segmentation and object detection. No post - processing was applied after estimation.

For semantic segmentation, we use the F1 - score to evaluate the classification of each pixel. For object detection, we assess performance based on the Generalized Intersection over Union (GIoU) and the accuracy of the predicted bounding box class. The evaluation is conducted on 500 pages from the PDF evaluation dataset of DocLayNet.

Benchmark

Now, let's compare the performance of this model with other models.

📄 License

This model is licensed under the apache - 2.0 license.

📖 Citation

@online{DeDetrLay,
  AUTHOR = {Cyrile Delestre},
  URL = {https://huggingface.co/cmarkea/detr-layout-detection},
  YEAR = {2024},
  KEYWORDS = {Image Processing ; Transformers ; Layout},
}