Math Fasttext Classifier

🚀 Maths FastText Classifier

This Maths FastText Classifier is part of a collection for curating pretraining datasets. It can classify text into Maths or Others, and is useful for LLM pretraining data curation to enhance mathematical capabilities.

🚀 Quick Start

This Maths FastText Classifier is designed to classify text into two categories: Maths and Others. It's trained on a large dataset and can be used for LLM pretraining data curation. You can access the Dataset here.

This classifier is part of the fasttext classifier collection for curating pretraining datasets. It classifies text into Maths or Others. The model is trained over 1.6M records, a 50:50 mix of maths and non - maths data from websites, achieving a test F1 score of 0.99. It intentionally upsamples maths data.

The classifier can be used for LLM pretraining data curation to enhance mathematical capabilities. It's ultra - fast ⚡ with a throughput of ~2000 doc/s on CPU.

Don't underestimate the "old" fasttext classifier! It's a good and scalable solution. For example, QWEN2.5 - MATH uses fasttext to curate pretraining data, though its classifier is not open - sourced.

✨ Features

• Classifies text into Maths or Others.
• Trained on 1.6M records with a 50:50 mix of maths and non - maths data.
• Achieved a test F1 score of 0.99.
• Ultra - fast with a throughput of ~2000 doc/s on CPU.
• Useful for LLM pretraining data curation.

📦 Installation

The installation mainly involves downloading the model from the Hugging Face Hub. The code in the usage section will handle the download process automatically.

💻 Usage Examples

Basic Usage

from typing import List
import re
from huggingface_hub import hf_hub_download
import fasttext


model_hf = fasttext.load_model(hf_hub_download("kenhktsui/maths-fasttext-classifier", "model.bin"))


def replace_newlines(text: str) -> str:
  return re.sub("\n+", " ", text)


def predict(text_list: List[str]) -> List[dict]:
  text_list = [replace_newlines(text) for text in text_list]
  pred = model_hf.predict(text_list)
  return [{"label": l[0].lstrip("__label__"), "score": s[0]}
           for l, s in zip(*pred)]


predict([
  """This is a lightning fast model, which can classify at throughtput of 2000 doc/s with CPU""",
  """Differential geometry is a mathematical discipline that studies the geometry of smooth shapes and smooth spaces, otherwise known as smooth manifolds. It uses the techniques of single variable calculus, vector calculus, linear algebra and multilinear algebra.""",
  """Given $p$: $|4x - 3|\leqslant 1$ and $q$: $x^{2}-(2a + 1)x + a^{2}+a\leqslant 0$, find the range of values for $a$ if $p$ is a necessary but not sufficient condition for $q$."""
])
# [{'label': 'Others', 'score': 1.00000834},
# {'label': 'Maths', 'score': 0.99995351},
# {'label': 'Maths', 'score': 0.99801832}]

📚 Documentation

Evaluation

              precision    recall  f1 - score   support

       Maths       0.99      0.98      0.99    200000
      Others       0.98      0.99      0.99    200000

    accuracy                           0.99    400000
   macro avg       0.99      0.99      0.99    400000
weighted avg       0.99      0.99      0.99    400000

⚠️ Important Note

The classifier does not handle short text well, which might not be surprising.

📄 License

This project is licensed under the MIT license.