PP FormulaNet Plus M

🚀 PP-FormulaNet_plus-M

PP-FormulaNet_plus-M 是由 PaddleOCR 团队开发的公式识别模型，在原有的 PP-FormulaNet 基础上进行了增强。该模型在训练时使用了更丰富多样的公式数据集，包括中文论文、专业书籍、教科书、试卷和数学期刊等来源，显著提升了模型的识别能力。PP-FormulaNet_plus 包含多个版本：L、M 和 S，其中 PP-FormulaNet_plus-M 增加了对中文公式的支持，并将公式预测的最大令牌数从 1024 增加到 2560，大大提高了对复杂公式的识别性能。

🚀 快速开始

📦 安装指南

1. 安装 PaddlePaddle

请参考以下命令，使用 pip 安装 PaddlePaddle：

# 适用于 CUDA11.8
python -m pip install paddlepaddle-gpu==3.0.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/

# 适用于 CUDA12.6
python -m pip install paddlepaddle-gpu==3.0.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu126/

# 适用于 CPU
python -m pip install paddlepaddle==3.0.0 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/

PaddlePaddle 安装的详细信息，请参考 PaddlePaddle 官方网站。

2. 安装 PaddleOCR

从 PyPI 安装最新版本的 PaddleOCR 推理包：

python -m pip install paddleocr

💻 使用示例

基础用法

你可以通过一条命令快速体验该功能：

paddleocr formula_recognition \
    --model_name PP-FormulaNet_plus-M  \
    -i https://cdn-uploads.huggingface.co/production/uploads/68493f0616e67d38f02f138a/4kkIUGxXMGozIg6U1BIxZ.png

你也可以将公式识别模块的模型推理集成到你的项目中。在运行以下代码之前，请将 示例图像 下载到本地：

from paddleocr import FormulaRecognition
model = FormulaRecognition(model_name="PP-FormulaNet_plus-M")
output = model.predict(input="4kkIUGxXMGozIg6U1BIxZ.png", batch_size=1)
for res in output:
    res.print()
    res.save_to_img(save_path="./output/")
    res.save_to_json(save_path="./output/res.json")

运行后，得到的结果如下：

{'res': {'input_path': '4kkIUGxXMGozIg6U1BIxZ.png', 'page_index': None, 'rec_formula': '\\zeta_{0}(\\nu)=-\\frac{\\nu\\varrho^{-2\\nu}}{\\pi}\\int_{\\mu}^{\\infty}d\\omega\\int_{C_{+}}d z\\frac{2z^{2}}{(z^{2}+\\omega^{2})^{\\nu+1}}\\breve{\\Psi}(\\omega;z)e^{i\\epsilon z}\\quad,'}}

⚠️ 重要提示

如果需要可视化公式识别模块，必须通过运行以下命令安装 LaTeX 渲染环境。目前，仅支持在 Ubuntu 上进行可视化，其他环境暂不支持。对于复杂公式，LaTeX 结果可能包含高级表示，在 Markdown 或类似环境中可能无法成功渲染：

sudo apt-get update
sudo apt-get install texlive texlive-latex-base texlive-xetex latex-cjk-all texlive-latex-extra -y

可视化后的图像如下：

使用命令和参数说明的详细信息，请参考 文档。

高级用法

单个模型的能力是有限的，但由多个模型组成的管道可以提供更强的能力来解决现实场景中的难题。

公式识别管道

公式识别管道旨在通过从图像中提取公式信息并以 LaTeX 源代码格式输出，来解决公式识别任务。该管道包含 4 个模块：

• 文档图像方向分类模块（可选）
• 文本图像校正模块（可选）
• 布局检测模块（可选）
• 公式识别模块

运行以下命令，快速体验公式识别管道。在运行以下代码之前，请将 示例图像 下载到本地：

paddleocr formula_recognition_pipeline -i https://cdn-uploads.huggingface.co/production/uploads/68493f0616e67d38f02f138a/4HrLNUf2yKGI8CwN9axpt.png \
    --formula_recognition_model_name PP-FormulaNet_plus-M  \
    --save_path ./output \
    --device gpu:0 

结果将打印到终端：

{'res': {'input_path': '/root/.paddlex/predict_input/4HrLNUf2yKGI8CwN9axpt.png', 'page_index': None, 'model_settings': {'use_doc_preprocessor': True, 'use_layout_detection': True}, 'doc_preprocessor_res': {'input_path': None, 'page_index': None, 'model_settings': {'use_doc_orientation_classify': True, 'use_doc_unwarping': True}, 'angle': 0}, 'layout_det_res': {'input_path': None, 'page_index': None, 'boxes': [{'cls_id': 2, 'label': 'text', 'score': 0.9855162501335144, 'coordinate': [90.5582, 1086.7775, 658.8992, 1553.267]}, {'cls_id': 2, 'label': 'text', 'score': 0.9814791679382324, 'coordinate': [93.042145, 127.992386, 664.8606, 396.60297]}, {'cls_id': 2, 'label': 'text', 'score': 0.9767233729362488, 'coordinate': [698.44617, 591.048, 1293.3668, 748.28625]}, {'cls_id': 2, 'label': 'text', 'score': 0.9712724089622498, 'coordinate': [701.4879, 286.62286, 1299.0151, 391.8841]}, {'cls_id': 2, 'label': 'text', 'score': 0.9708836078643799, 'coordinate': [697.0071, 751.9401, 1290.2227, 883.6447]}, {'cls_id': 2, 'label': 'text', 'score': 0.9688520431518555, 'coordinate': [704.01917, 79.636734, 1304.7367, 187.96138]}, {'cls_id': 2, 'label': 'text', 'score': 0.9683284163475037, 'coordinate': [93.07703, 799.36597, 660.6864, 902.0364]}, {'cls_id': 7, 'label': 'formula', 'score': 0.9660061597824097, 'coordinate': [728.5604, 440.9317, 1224.097, 570.8568]}, {'cls_id': 7, 'label': 'formula', 'score': 0.9615049958229065, 'coordinate': [723.025, 1333.5005, 1257.1569, 1468.0688]}, {'cls_id': 7, 'label': 'formula', 'score': 0.961004376411438, 'coordinate': [777.5282, 207.88376, 1222.9387, 267.32993]}, {'cls_id': 7, 'label': 'formula', 'score': 0.9609803557395935, 'coordinate': [756.4403, 1211.3208, 1188.0408, 1268.2334]}, {'cls_id': 2, 'label': 'text', 'score': 0.959402322769165, 'coordinate': [697.5221, 957.6737, 1288.6223, 1033.5424]}, {'cls_id': 2, 'label': 'text', 'score': 0.9592350125312805, 'coordinate': [691.3296, 1511.7983, 1282.0968, 1642.5952]}, {'cls_id': 7, 'label': 'formula', 'score': 0.9590734839439392, 'coordinate': [153.89197, 924.2169, 601.09546, 1036.9056]}, {'cls_id': 2, 'label': 'text', 'score': 0.9582054615020752, 'coordinate': [87.024506, 1557.2972, 655.9558, 1632.701]}, {'cls_id': 7, 'label': 'formula', 'score': 0.9579665064811707, 'coordinate': [810.86975, 1057.1064, 1175.1078, 1117.6572]}, {'cls_id': 7, 'label': 'formula', 'score': 0.9557791352272034, 'coordinate': [165.2392, 557.844, 598.2451, 614.3662]}, {'cls_id': 7, 'label': 'formula', 'score': 0.9539064764976501, 'coordinate': [116.46484, 713.8796, 614.2107, 774.029]}, {'cls_id': 2, 'label': 'text', 'score': 0.9520670175552368, 'coordinate': [96.68561, 478.32416, 662.5837, 536.60223]}, {'cls_id': 2, 'label': 'text', 'score': 0.9442729949951172, 'coordinate': [96.14572, 639.16113, 661.80334, 692.4822]}, {'cls_id': 2, 'label': 'text', 'score': 0.940317690372467, 'coordinate': [695.9426, 1138.6841, 1286.7327, 1188.0151]}, {'cls_id': 7, 'label': 'formula', 'score': 0.9249900579452515, 'coordinate': [852.94556, 908.64923, 1131.185, 933.8394]}, {'cls_id': 7, 'label': 'formula', 'score': 0.9248911142349243, 'coordinate': [195.27357, 424.8133, 567.68335, 451.1208]}, {'cls_id': 17, 'label': 'formula_number', 'score': 0.9173402786254883, 'coordinate': [1246.2461, 1079.063, 1286.333, 1104.3276]}, {'cls_id': 17, 'label': 'formula_number', 'score': 0.9168799519538879, 'coordinate': [1246.8928, 908.664, 1288.1958, 934.6163]}, {'cls_id': 17, 'label': 'formula_number', 'score': 0.915979266166687, 'coordinate': [1247.0377, 1229.1577, 1287.0939, 1254.9792]}, {'cls_id': 17, 'label': 'formula_number', 'score': 0.9086456894874573, 'coordinate': [1252.8517, 492.109, 1294.6124, 518.47156]}, {'cls_id': 17, 'label': 'formula_number', 'score': 0.9016352891921997, 'coordinate': [1242.1753, 1473.7004, 1283.019, 1498.6516]}, {'cls_id': 17, 'label': 'formula_number', 'score': 0.9000396728515625, 'coordinate': [1269.8044, 220.35562, 1299.8611, 247.01315]}, {'cls_id': 7, 'label': 'formula', 'score': 0.8966289758682251, 'coordinate': [95.999916, 235.48334, 295.44852, 265.59302]}, {'cls_id': 2, 'label': 'text', 'score': 0.8954761028289795, 'coordinate': [696.8688, 1286.2268, 1083.3927, 1310.8733]}, {'cls_id': 7, 'label': 'formula', 'score': 0.8951668739318848, 'coordinate': [166.62683, 129.18127, 511.6576, 156.2976]}, {'cls_id': 2, 'label': 'text', 'score': 0.8934891819953918, 'coordinate': [725.67053, 396.18787, 1263.0408, 422.78894]}, {'cls_id': 17, 'label': 'formula_number', 'score': 0.892305314540863, 'coordinate': [634.14246, 427.77844, 661.17773, 454.10535]}, {'cls_id': 2, 'label': 'text', 'score': 0.8891267776489258, 'coordinate': [94.483185, 1058.7578, 441.93515, 1082.4932]}, {'cls_id': 17, 'label': 'formula_number', 'score': 0.8877044320106506, 'coordinate': [630.4214, 939.2977, 657.7117, 965.36]}, {'cls_id': 17, 'label': 'formula_number', 'score': 0.8832477927207947, 'coordinate': [630.59216, 1000.9552, 657.4265, 1026.2094]}, {'cls_id': 17, 'label': 'formula_number', 'score': 0.8769293427467346, 'coordinate': [634.1151, 575.3828, 660.59314, 601.1638]}, {'cls_id': 7, 'label': 'formula', 'score': 0.8733161091804504, 'coordinate': [95.2839, 1320.377, 264.92313, 1345.8511]}, {'cls_id': 17, 'label': 'formula_number', 'score': 0.8703839182853699, 'coordinate': [633.8277, 730.3137, 659.84564, 755.55347]}, {'cls_id': 7, 'label': 'formula', 'score': 0.8392633199691772, 'coordinate': [365.18652, 268.2967, 515.7993, 296.06952]}, {'cls_id': 7, 'label': 'formula', 'score': 0.8316442370414734, 'coordinate': [1090.5394, 1599.1625, 1276.672, 1622.1669]}, {'cls_id': 7, 'label': 'formula', 'score': 0.817266583442688, 'coordinate': [246.17564, 161.22665, 314.3683, 186.41296]}, {'cls_id': 3, 'label': 'number', 'score': 0.8043113350868225, 'coordinate': [1297.4021, 7.143423, 1310.6084, 27.74441]}, {'cls_id': 7, 'label': 'formula', 'score': 0.797702968120575, 'coordinate': [538.4617, 478.0895, 661.8972, 508.51443]}, {'cls_id': 7, 'label': 'formula', 'score': 0.7646714448928833, 'coordinate': [916.5115, 1618.5238, 1009.6382, 1640.8141]}, {'cls_id': 7, 'label': 'formula', 'score': 0.7432729005813599, 'coordinate': [694.83765, 1612.2528, 861.0532, 1635.9652]}, {'cls_id': 7, 'label': 'formula', 'score': 0.7072135806083679, 'coordinate': [99.70873, 508.2096, 254.92291, 535.7488]}, {'cls_id': 7, 'label': 'formula', 'score': 0.6994448304176331, 'coordinate': [696.79846, 1561.4436, 899.79236, 1586.7415]}, {'cls_id': 7, 'label': 'formula', 'score': 0.6704866886138916, 'coordinate': [1117.0674, 1572.0345, 1191.5293, 1594.7426]}, {'cls_id': 7, 'label': 'formula', 'score': 0.6333382725715637, 'coordinate': [577.34186, 1274.4236, 602.5528, 1296.696]}, {'cls_id': 7, 'label': 'formula', 'score': 0.621163010597229, 'coordinate': [175.29213, 349.82397, 241.25047, 376.66553]}, {'cls_id': 7, 'label': 'formula', 'score': 0.6146791577339172, 'coordinate': [773.0735, 595.1659, 800.43823, 617.38635]}, {'cls_id': 7, 'label': 'formula', 'score': 0.6107904314994812, 'coordinate': [706.67114, 316.8644, 736.7178, 339.93387]}, {'cls_id': 7, 'label': 'formula', 'score': 0.5521712899208069, 'coordinate': [1263.9779, 314.65396, 1292.8451, 337.40207]}, {'cls_id': 7, 'label': 'formula', 'score': 0.5341379046440125, 'coordinate': [1219.2937, 316.60284, 1243.9319, 339.71375]}, {'cls_id': 7, 'label': 'formula', 'score': 0.520746111869812, 'coordinate': [254.65915, 323.65292, 326.58456, 349.53452]}, {'cls_id': 7, 'label': 'formula', 'score': 0.5011299848556519, 'coordinate': [255.84404, 1350.6619, 301.73444, 1375.5315]}]}, 'formula_res_list': [{'rec_formula': '\\begin{aligned}{\\psi_{0}(M)-\\psi_{}(M,z)=}&{{}\\frac{(1-\\epsilon_{r})}{\\epsilon_{r}}\\frac{\\lambda^{2}c^{2}}{t_{\\operatorname{E}}^{2}\\operatorname{l n}(10)}\\times}\\\\ {}&{{}\\int_{0}^{z}d z^{\\prime}\\frac{d t}{d z^{\\prime}}\\left.\\frac{\\partial\\phi}{\\partial L}\\right|_{L=\\lambda M c^{2}/t_{\\operatorname{E}}},}\\\\ \\end{aligned}', 'formula_region_id': 1, 'dt_polys': ([728.5604, 440.9317, 1224.097, 570.8568],)}, {'rec_formula': '\\begin{aligned}{p(\\operatorname{l o g}_{10}}&{{}M|\\operatorname{l o g}_{10}\\sigma)=\\frac{1}{\\sqrt{2\\pi}\\epsilon_{0}}}\\\\ {}&{{}\\times\\operatorname{e x p}\\left[-\\frac{1}{2}\\left(\\frac{\\operatorname{l o g}_{10}M-a_{\\bullet}-b_{\\bullet}\\operatorname{l o g}_{10}\\sigma}{\\epsilon_{0}}\\right)^{2}\\right].}\\\\ \\end{aligned}', 'formula_region_id': 2, 'dt_polys': ([723.025, 1333.5005, 1257.1569, 1468.0688],)}, {'rec_formula': '\\phi(L)\\equiv\\frac{d n}{d\\operatorname{l o g}_{10}L}=\\frac{\\phi_{*}}{(L/L_{*})^{\\gamma_{1}}+(L/L_{*})^{\\gamma_{2}}}.', 'formula_region_id': 3, 'dt_polys': ([777.5282, 207.88376, 1222.9387, 267.32993],)}, {'rec_formula': '\\psi_{0}(M)=\\int d\\sigma\\frac{p(\\operatorname{l o g}_{10}M|\\operatorname{l o g}_{10}\\sigma)}{M\\operatorname{l o g}(10)}\\frac{d n}{d\\sigma}(\\sigma),', 'formula_region_id': 4, 'dt_polys': ([756.4403, 1211.3208, 1188.0408, 1268.2334],)}, {'rec_formula': '\\begin{aligned}{\\rho_{\\operatorname{B H}}}&{{}=\\int d M\\psi(M)M}\\\\ {}&{{}=\\frac{1-\\epsilon_{r}}{\\epsilon_{r}c^{2}}\\int_{0}^{\\infty}d z\\frac{d t}{d z}\\int d\\operatorname{l o g}_{10}L\\phi(L,z)L,}\\\\ \\end{aligned}', 'formula_region_id': 5, 'dt_polys': ([153.89197, 924.2169, 601.09546, 1036.9056],)}, {'rec_formula': '\\frac{d n}{d\\sigma}d\\sigma=\\psi_{*}\\left(\\frac{\\sigma}{\\sigma_{*}}\\right)^{\\alpha}\\frac{e^{-(\\sigma/\\sigma_{*})^{\\beta}}}{\\Gamma(\\alpha/\\beta)}\\beta\\frac{d\\sigma}{\\sigma}.', 'formula_region_id': 6, 'dt_polys': ([810.86975, 1057.1064, 1175.1078, 1117.6572],)}, {'rec_formula': '\\langle\\dot{M}(M,t)\\rangle\\psi(M,t)=\\frac{(1-\\epsilon_{r})}{\\epsilon_{r}c^{2}\\operatorname{l n}(10)}\\phi(L,t)\\frac{d L}{d M}.', 'formula_region_id': 7, 'dt_polys': ([165.2392, 557.844, 598.2451, 614.3662],)}, {'rec_formula': '\\frac{\\partial\\psi}{\\partial t}(M,t)+\\frac{(1-\\epsilon_{r})}{\\epsilon_{r}}\\frac{\\lambda^{2}c^{2}}{t_{\\operatorname{E}}^{2}\\operatorname{l n}(10)}\\left.\\frac{\\partial\\phi}{\\partial L}\\right|_{L=\\lambda M c^{2}/t_{\\operatorname{E}}}=0,', 'formula_region_id': 8, 'dt_polys': ([116.46484, 713.8796, 614.2107, 774.029],)}, {'rec_formula': '\\log_{10}M=a_{\\bullet}+b_{\\bullet}\\log_{10}X.', 'formula_region_id': 9, 'dt_polys': ([852.94556, 908.64923, 1131.185, 933.8394],)}, {'rec_formula': '\\phi(L,t)d\\operatorname{l o g}_{10}L=\\delta(M,t)\\psi(M,t)d M.', 'formula_region_id': 10, 'dt_polys': ([195.27357, 424.8133, 567.68335, 451.1208],)}, {'rec_formula': '\\dot{M}=(1-\\epsilon_{r})\\dot{M}_{\\mathrm{a c c}}^{\\mathrm{~\\tiny~\\cdot~}}', 'formula_region_id': 11, 'dt_polys': ([95.999916, 235.48334, 295.44852, 265.59302],)}, {'rec_formula': 't_{E}=\\sigma_{T}c/4\\pi G m_{p}=4.5\\times10^{8}\\mathrm{y r}', 'formula_region_id': 12, 'dt_polys': ([166.62683, 129.18127, 511.6576, 156.2976],)}, {'rec_formula': 'M_{*}=L_{*}t_{E}/\\tilde{\\lambda}c^{2}', 'formula_region_id': 13, 'dt_polys': ([95.2839, 1320.377, 264.92313, 1345.8511],)}, {'rec_formula': '\\phi(L,t)d\\operatorname{l o g}_{10}L', 'formula_region_id': 14, 'dt_polys': ([365.18652, 268.2967, 515.7993, 296.06952],)}, {'rec_formula': 'a_{\\bullet}=8.32\\pm0.05', 'formula_region_id': 15, 'dt_polys': ([1090.5394, 1599.1625, 1276.672, 1622.1669],)}, {'rec_formula': '\\epsilon_{r}\\dot{M}_{\\operatorname{a c c}}', 'formula_region_id': 16, 'dt_polys': ([246.17564, 161.22665, 314.3683, 186.41296],)}, {'rec_formula': '\\langle\\dot{M}(M,t)\\rangle=', 'formula_region_id': 17, 'dt_polys': ([538.4617, 478.0895, 661.8972, 508.51443],)}, {'rec_formula': '\\epsilon_{0}=0.38', 'formula_region_id': 18, 'dt_polys': ([916.5115, 1618.5238, 1009.6382, 1640.8141],)}, {'rec_formula': 'b_{\\bullet}=5.64\\dot{\\pm}\\dot{0.32}', 'formula_region_id': 19, 'dt_polys': ([694.83765, 1612.2528, 861.0532, 1635.9652],)}, {'rec_formula': '\\delta(M,t)\\dot{M}(M,t)', 'formula_region_id': 20, 'dt_polys': ([99.70873, 508.2096, 254.92291, 535.7488],)}, {'rec_formula': 'X=\\sigma/200\\mathrm{k m}\\mathrm{s^{-1}}', 'formula_region_id': 21, 'dt_polys': ([696.79846, 1561.4436, 899.79236, 1586.7415],)}, {'rec_formula': 'M-\\sigma', 'formula_region_id': 22, 'dt_polys': ([1117.0674, 1572.0345, 1191.5293, 1594.7426],)}, {'rec_formula': 'L_{*}', 'formula_region_id': 23, 'dt_polys': ([577.34186, 1274.4236, 602.5528, 1296.696],)}, {'rec_formula': '\\phi(L,t)', 'formula_region_id': 24, 'dt_polys': ([175.29213, 349.82397, 241.25047, 376.66553],)}, {'rec_formula': '\\psi_{0}', 'formula_region_id': 25, 'dt_polys': ([773.0735, 595.1659, 800.43823, 617.38635],)}, {'rec_formula': '\\mathrm{A^{\\prime\\prime}}', 'formula_region_id': 26, 'dt_polys': ([706.67114, 316.8644, 736.7178, 339.93387],)}, {'rec_formula': 'L_{*}', 'formula_region_id': 27, 'dt_polys': ([1263.9779, 314.65396, 1292.8451, 337.40207],)}, {'rec_formula': '\\phi_{*}', 'formula_region_id': 28, 'dt_polys': ([1219.2937, 316.60284, 1243.9319, 339.71375],)}, {'rec_formula': '\\delta(M,t)', 'formula_region_id': 29, 'dt_polys': ([254.65915, 323.65292, 326.58456, 349.53452],)}, {'rec_formula': '\\phi(L)', 'formula_region_id': 30, 'dt_polys': ([255.84404, 1350.6619, 301.73444, 1375.5315],)}]}}

如果指定了 save_path，可视化结果将保存到 save_path 下。可视化输出如下：

命令行方法适用于快速体验，对于项目集成，也只需要几行代码：

from paddleocr import FormulaRecognitionPipeline

pipeline = FormulaRecognitionPipeline(formula_recognition_model_name="PP-FormulaNet_plus-M")
output = pipeline.predict("./4HrLNUf2yKGI8CwN9axpt.png")
for res in output:
    res

✨ 主要特性

• 丰富的训练数据：使用了包括中文论文、专业书籍、教科书、试卷和数学期刊等更丰富多样的公式数据集进行训练。
• 支持中文公式：PP-FormulaNet_plus-M 增加了对中文公式的支持。
• 更多的预测令牌数：将公式预测的最大令牌数从 1024 增加到 2560，提高了对复杂公式的识别性能。

📚 详细文档

模型性能指标

注：英文 BLEU 和中文 BLEU (%) 分别代表英文公式和中文公式的 BLEU 得分。英文公式的评估数据集包括来自 UniMERNet 的简单和复杂公式，以及 PaddleX 内部开发数据集的简单、中等和复杂公式。中文公式的评估数据集来自 PaddleX 内部开发的中文公式数据集。

📄 许可证

本项目采用 Apache-2.0 许可证。