Pix2text Mfr

🚀 Model Card: Pix2Text-MFR

A Mathematical Formula Recognition (MFR) model that converts images of mathematical formulas into LaTeX text representation.

🚀 Quick Start

This MFR model can be used to convert images of mathematical formulas into LaTeX text representation. You can choose different methods to use the model according to your needs.

✨ Features

• Utilizes the TrOCR architecture developed by Microsoft.
• Can convert input images of mathematical formulas into LaTeX text representation.
• Supports different usage methods, including direct use, using Pix2Text, and using a notebook.

📦 Installation

Method 2: Using Pix2Text

$ pip install pix2text>=1.1

💻 Usage Examples

Basic Usage

Method 1: Using the model Directly

#! pip install transformers>=4.37.0 pillow optimum[onnxruntime]
from PIL import Image
from transformers import TrOCRProcessor
from optimum.onnxruntime import ORTModelForVision2Seq

processor = TrOCRProcessor.from_pretrained('breezedeus/pix2text-mfr')
model = ORTModelForVision2Seq.from_pretrained('breezedeus/pix2text-mfr', use_cache=False)

image_fps = [
    'examples/example.jpg',
    'examples/42.png',
    'examples/0000186.png',
]
images = [Image.open(fp).convert('RGB') for fp in image_fps]
pixel_values = processor(images=images, return_tensors="pt").pixel_values
generated_ids = model.generate(pixel_values)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
print(f'generated_ids: {generated_ids}, \ngenerated text: {generated_text}')

Method 2: Using Pix2Text

#! pip install pix2text>=1.1

from pix2text import Pix2Text, merge_line_texts

image_fps = [
    'examples/example.jpg',
    'examples/42.png',
    'examples/0000186.png',
]
p2t = Pix2Text.from_config()
outs = p2t.recognize_formula(image_fps)  # recognize pure formula images

outs2 = p2t.recognize('examples/mixed.jpg', file_type='text_formula', return_text=True, save_analysis_res='mixed-out.jpg')  # recognize mixed images
print(outs2)

Method 3: Notebook

Just try Pix2Text with this notebook: https://github.com/breezedeus/Pix2Text/blob/main/pix2text_v1_1.ipynb.

📚 Documentation

• Pix2Text V1.0 New Release: The Best Open-Source Formula Recognition Model | Breezedeus.com
• Pix2Text (P2T) Github: breezedeus/pix2text
• Pix2Text Online Free Service: p2t.breezedeus.com
• Pix2Text Online Docs: Docs
• Pix2Text More: breezedeus.com/pix2text
• Pix2Text Discard: https://discord.gg/GgD87WM8Tf

🔧 Technical Details

This MFR model utilizes the TrOCR architecture developed by Microsoft, starting with its initial values and retrained using a dataset of mathematical formula images. The resulting MFR model can be used to convert images of mathematical formulas into LaTeX text representation. More detailed can be found: Pix2Text V1.0 New Release: The Best Open-Source Formula Recognition Model | Breezedeus.com.

📄 License

This project is licensed under the MIT license.

📊 Performance

The test dataset includes 485 images, which are derived from real data uploaded by users on the Pix2Text Online Service. The following is the Character Error Rates (CER) of various models on this test dataset:

As can be seen from the figure, the Pix2Text V1.0 MFR open-source free version model has significantly outperformed the previous versions of the paid model. Moreover, compared to the V1.0 MFR open-source free model, the precision of the Pix2Text V1.0 MFR paid model has been further improved.

⚠️ Important Note

Texify is more suited for recognizing images with standard formatting. It performs poorly in recognizing images containing single letters. This is the main reason why Texify's performance on this test dataset is inferior to that of Latex-OCR.

💬 Feedback

Welcome to contact the author Breezedeus if you have any questions or comments about the model.