Layoutlmv3 Base Mpdocvqa

🚀 LayoutLMv3 base fine-tuned on MP-DocVQA

This is a pre - trained LayoutLMv3 model from Microsoft hub, which has been fine - tuned on the Multipage DocVQA (MP - DocVQA) dataset. It was used as a baseline in Hierarchical multimodal transformers for Multi - Page DocVQA.

Model Information

🚀 Quick Start

This model is a fine - tuned version of the pre - trained LayoutLMv3 from Microsoft hub on the Multipage DocVQA (MP - DocVQA) dataset. It was used as a baseline in Hierarchical multimodal transformers for Multi - Page DocVQA.

• Results on the MP - DocVQA dataset are reported in Table 2.
• Training hyperparameters can be found in Table 8 of Appendix D.

💻 Usage Examples

Basic Usage

import torch
from transformers import LayoutLMv3Processor, LayoutLMv3ForQuestionAnswering

processor = LayoutLMv3Processor.from_pretrained("rubentito/layoutlmv3-base-mpdocvqa", apply_ocr=False)
model = LayoutLMv3ForQuestionAnswering.from_pretrained("rubentito/layoutlmv3-base-mpdocvqa")

image = Image.open("example.jpg").convert("RGB")
question = "Is this a question?"
context = ["Example"]
boxes = [0, 0, 1000, 1000]  # This is an example bounding box covering the whole image.
document_encoding = processor(image, question, context, boxes=boxes, return_tensors="pt")
outputs = model(**document_encoding)

# Get the answer
start_idx = torch.argmax(outputs.start_logits, axis=1)
end_idx = torch.argmax(outputs.end_logits, axis=1)
answers = self.processor.tokenizer.decode(input_tokens[start_idx: end_idx+1]).strip()

📚 Documentation

Metrics

Average Normalized Levenshtein Similarity (ANLS)

The standard metric for text - based VQA tasks (ST - VQA and DocVQA). It evaluates the method's reasoning capabilities while smoothly penalizes OCR recognition errors. Check Scene Text Visual Question Answering for detailed information.

Answer Page Prediction Accuracy (APPA)

In the MP - DocVQA task, the models can provide the index of the page where the information required to answer the question is located. For this subtask, accuracy is used to evaluate the predictions: i.e., if the predicted page is correct or not. Check Hierarchical multimodal transformers for Multi - Page DocVQA for detailed information.

Model results

Extended experimentation can be found in Table 2 of Hierarchical multimodal transformers for Multi - Page DocVQA. You can also check the live leaderboard at the RRC Portal.

📄 License

The model uses the cc - by - nc - sa - 4.0 license.

📖 Citation Information

@article{tito2022hierarchical,
  title={Hierarchical multimodal transformers for Multi-Page DocVQA},
  author={Tito, Rub{\`e}n and Karatzas, Dimosthenis and Valveny, Ernest},
  journal={arXiv preprint arXiv:2212.05935},
  year={2022}
}