Modernbert Content Regression

🚀 ModernBERT Engagement Content Regression

This project explores using ModernBERT for the text regression task of predicting engagement metrics for text content, specifically the click - through rate (CTR) of email text content.

🚀 Quick Start

Install dependencies and activate venv

uv sync
source .venv/bin/activate

The following values need to be defined in the .env file:

• HUGGINGFACE_TOKEN

Run notebook for model fitting

uv run --with jupyter jupyter lab

✨ Features

This project explores using ModernBERT for text regression to predict the click - through rate (CTR) of email text content. It includes hyperparameter tuning of ModernBERT and comparison with a benchmark model.

📦 Installation

Install dependencies and activate venv

uv sync
source .venv/bin/activate

Ensure the following values are defined in the .env file:

• HUGGINGFACE_TOKEN

💻 Usage Examples

Run notebook for model fitting

uv run --with jupyter jupyter lab

📚 Documentation

What is this?

This project explores using ModernBERT for the text regression task of predicting engagement metrics for text content. In this case, we predict the click - through rate (CTR) of email text content.

We will explore ModernBERT's hyperparameter tuning and how to use it for regression. We will also compare the results to a benchmark model.

This type of task is complex, as we can remember the quote:

Half my advertising is wasted; the trouble is, I don't know which half -John Wanamaker

In this experiment, we exclude other relevant factors, such as the time the email is sent, the day of the week, the recipient, etc.

Links for the project:

• Model - ModernBERT - Engagement - Content - Regression
• Training notebook - Training Notebook
• Demo - Demo Space

This work is indebted to the work of many community members and blog posts:

• ModernBERT Announcement
• Fine - tune classifier with ModernBERT in 2025
• How to set up Trainer for a regression
• Additional thanks to the creators of ModernBERT!

Our dataset

We will be using a dataset of 548 emails where we have the text of the email and the CTR we are trying to predict as labels.

We look forward to ModernBERT's improvements, allowing us to fine - tune models for each potential user’s email dataset. The variability of email data and its small size pose interesting regression challenges.

Benchmarking

We will start by using the Catboost library as a simple benchmark for text regression. For both the benchmark and the ModernBERT run, we are using 'rmse' as the metric. We receive the following results:

ModernBERT Model Performance

After running hyperparameter tuning for ModernBERT, we get the following results:

We see improvements in all metrics except for SMAPE. We believe that ModernBERT would scale even better with a larger dataset; as 500 examples is very low for fine - tuning, and we are thus happy with the performance of this evaluation.

Who are we?

At Forecast.ing we are building a platform to help users create more enriching content by automatically researching trends and generating campaign ideas with AgenticAI. We generate the content and then create fine - tuned scores of how likely we think that content will succeed.

🔧 Technical Details

This project uses ModernBERT for text regression. We first use the Catboost library as a benchmark for text regression. For both the benchmark and the ModernBERT run, we use 'rmse' as the metric. We also perform hyperparameter tuning on ModernBERT.

📄 License

This project is under the Apache - 2.0 license.

Conclusion

We see that ModernBERT is a powerful model for text regression. We believe that with a larger dataset, we would see even better results. We are excited to see the future of ModernBERT and how it will be used for text regression. If interested, you can contact us at robin@forecast.ing