Donut Base Finetuned Invoices

🚀 Donut finetuned on invoices

This project is based on the Donut base model, fine - tuned on invoices to extract key information from multi - language invoice documents.

🚀 Quick Start

The model is based on the Donut base model introduced in the paper OCR - free Document Understanding Transformer by Geewok et al. and first released in this repository.

The model was trained with a few thousand of annotated invoices and non - invoices (for non - invoices, the doctype will be 'Other'). These documents span across different countries and languages and are all one - page only. Unfortunately, the dataset is proprietary. The model is set to an input resolution of 1280x1920 pixels. So, any sample with a dpi higher than 150 has no added value.

It was trained for about 4 hours on a NVIDIA RTX A4000 for 20k steps, and the final val_metric was 0.03413819904382196.

The following indexes were included in the train set:

• DocType
• Currency
• DocumentDate
• GrossAmount
• InvoiceNumber
• NetAmount
• TaxAmount
• OrderNumber
• CreditorCountry

Demo space can be found here

✨ Features

• Multilingual Support: Trained on invoices from different countries and languages, suitable for multi - language invoice processing.
• Key Information Extraction: Can extract key information such as DocType, Currency, etc. from invoices.

📚 Documentation

Model description

Donut consists of a vision encoder (Swin Transformer) and a text decoder (BART). Given an image, the encoder first encodes the image into a tensor of embeddings (of shape batch_size, seq_len, hidden_size). After that, the decoder autoregressively generates text, conditioned on the encoding of the encoder.

Intended uses & limitations

This model is meant as a research in how well it fares with multilanguage invoices. See my observations in the demo space.

How to use

Look at the documentation which includes code examples.

📄 License

This model is licensed under cc - by - nc - sa - 4.0.