Open-source text classification model llm-data-textbook-quality-fasttext-classifier-v1, accurately filter textbook-level data!

Llm Data Textbook Quality Fasttext Classifier V1

Developed by kenhktsui

A text classification model built on fasttext, used to determine whether text meets textbook-level data quality, serving as a data filtering tool for large language model training.

Text Classification EnglishOpen Source License:MIT #Textbook-level data filtering #High-speed text classification #LLM data preprocessing

Downloads 35

Release Time : 4/28/2024

Model Overview

This model is an optimized version of llm-data-textbook-quality-classifier-v1, achieving not only a higher F1 score but also the capability to classify over 2000 samples per second on a CPU.

Model Features

High performance

Capable of classifying over 2000 samples per second on a CPU, suitable for large-scale data processing.

High accuracy

Achieves an F1 score of 0.8695 on the training set and 0.8485 on the test set, demonstrating excellent performance.

Textbook-level quality detection

Specifically optimized for textbook-level data quality, effectively filtering high-quality training data.

Model Capabilities

Text quality classification

Data filtering

Large-scale text processing

Use Cases

Data preprocessing

Large language model training data filtering

Use this model to filter high-quality textbook-level data before training large language models.

Improves model training effectiveness and generation quality

Content quality assessment

Educational content quality assessment

Evaluate whether educational texts meet textbook-level quality standards.

Helps identify high-quality educational content

🚀 llm-data-textbook-quality-fasttext-classifier-v1

This model, built on fasttext, is an optimized version of llm-data-textbook-quality-classifier-v1. It can classify whether a text is of textbook-quality data, achieving a high F1 score and classifying over 2000 examples per second on CPU. It can serve as a filter for data curation during LLM training.

✨ Features

Built on fasttext, an optimized version of the previous model.
Achieves a high F1 score.
Can classify more than 2000 examples per second on CPU.
Capable of classifying if a text is of textbook quality data, useful for data curation in LLM training.

📦 Installation

No specific installation steps provided in the original README.

💻 Usage Examples

Basic Usage

from typing import List
import re
from huggingface_hub import hf_hub_download
import fasttext


model = fasttext.load_model(hf_hub_download("kenhktsui/llm-data-textbook-quality-fasttext-classifer-v1", "model.bin"))


def replace_newlines(text: str) -> str:
  return re.sub("\n+", " ", text)


def predict(text_list: List[str]) -> List[dict]:
  text_list = [replace_newlines(text) for text in text_list]
  pred = model.predict(text_list)
  return [{"label": l[0].lstrip("__label__"), "score": s[0]}
           for l, s in zip(*pred)]


predict(["Hi"])
# Output: [{'label': 'LOW_QUALITY', 'score': 1.00001}]

📚 Documentation

Model Performance

Dataset	F1 Score
Train	0.8695
Test	0.8485

Benchmark

Dataset	Sampling	Average Quality Score
nampdn-ai/tiny-orca-textbooks	Full	0.8350
nampdn-ai/tiny-textbooks	Full	0.7535
SciPhi/textbooks-are-all-you-need-lite	Full	0.7202
vikp/textbook_quality_programming	Full	0.5447
BEE-spoke-data/fineweb-100k_en-med	Full	0.4754
pszemraj/simple_wikipedia_LM	Full	0.4704
mattymchen/refinedweb-3m	Full	0.2963
JeanKaddour/minipile	Full	0.2562

Average Quality Score is defined as the average probability output of HIGH_QUALITY. The classifier aligns with the expectation. Textbook category scores the highest, reflecting the effectiveness of this model. Wikipedia scores lower because it is not textbook after all. Web scores the lowest.

📄 License

This model is licensed under the MIT license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご