Mgp Str Base

🚀 MGP-STR (base-sized model)

The MGP-STR base-sized model is trained on MJSynth and SynthText, offering effective scene text recognition capabilities.

🚀 Quick Start

The MGP-STR base-sized model is trained on MJSynth and SynthText. It was introduced in the paper Multi-Granularity Prediction for Scene Text Recognition and first released in this repository.

✨ Features

• It is a pure vision STR model, combining ViT and specially designed A^3 modules.
• Capable of multi - granularity predictions including character, subword, and word levels.
• Utilizes a simple and effective fusion strategy to merge multi - granularity predictions.

📚 Documentation

Model description

MGP-STR is a pure vision STR model, consisting of ViT and specially designed A^3 modules. The ViT module was initialized from the weights of DeiT - base, except the patch embedding model, due to the inconsistent input size.

Images (32x128) are presented to the model as a sequence of fixed - size patches (resolution 4x4), which are linearly embedded. One also adds absolute position embeddings before feeding the sequence to the layers of the ViT module. Next, the A^3 module selects a meaningful combination from the tokens of the ViT output and integrates them into one output token corresponding to a specific character. Moreover, subword classification heads based on the BPE A^3 module and WordPiece A^3 module are devised for subword predictions, so that the language information can be implicitly modeled. Finally, these multi - granularity predictions (character, subword and even word) are merged via a simple and effective fusion strategy.

Intended uses & limitations

You can use the raw model for optical character recognition (OCR) on text images. See the [model hub](https://huggingface.co/models?search=alibaba - damo/mgp - str) to look for fine - tuned versions on a task that interests you.

💻 Usage Examples

Basic Usage

from transformers import MgpstrProcessor, MgpstrForSceneTextRecognition
import requests
from PIL import Image

processor = MgpstrProcessor.from_pretrained('alibaba-damo/mgp-str-base')
model = MgpstrForSceneTextRecognition.from_pretrained('alibaba-damo/mgp-str-base')

# load image from the IIIT-5k dataset
url = "https://i.postimg.cc/ZKwLg2Gw/367-14.png"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")

pixel_values = processor(images=image, return_tensors="pt").pixel_values
outputs = model(pixel_values)

generated_text = processor.batch_decode(outputs.logits)['generated_text']

BibTeX entry and citation info

@inproceedings{ECCV2022mgp_str,
  title={Multi-Granularity Prediction for Scene Text Recognition},
  author={Peng Wang, Cheng Da, and Cong Yao},
  booktitle = {ECCV},
  year={2022}
}