Modernce Large Sts

🚀 ModernBERT Cross-Encoder: Semantic Similarity (STS)

This project offers a high - performing cross - encoder model for semantic similarity tasks. It compares two texts and outputs a score between 0 and 1. Leveraging the ModernBERT architecture and diverse training data, it provides an efficient and accurate solution for evaluating LLM outputs.

🚀 Quick Start

To use ModernCE for semantic similarity tasks, you can load the model with the Hugging Face sentence - transformers library:

from sentence_transformers import CrossEncoder

# Load ModernCE model
model = CrossEncoder("dleemiller/ModernCE-large-sts")

# Predict similarity scores for sentence pairs
sentence_pairs = [
    ("It's a wonderful day outside.", "It's so sunny today!"),
    ("It's a wonderful day outside.", "He drove to work earlier."),
]
scores = model.predict(sentence_pairs)

print(scores)  # Outputs: array([0.9184, 0.0123], dtype=float32)

Output

The model returns similarity scores in the range [0, 1], where higher scores indicate stronger semantic similarity.

✨ Features

• High performing: Achieves Pearson: 0.9256 and Spearman: 0.9215 on the STS - Benchmark test set.
• Efficient architecture: Based on the ModernBERT - large design (395M parameters), offering faster inference speeds.
• Extended context length: Processes sequences up to 8192 tokens, great for LLM output evals.
• Diversified training: Pretrained on dleemiller/wiki - sim and fine - tuned on sentence - transformers/stsb.

📊 Performance

📚 Documentation

Training Details

Pretraining

The model was pretrained on the pair - score - sampled subset of the [dleemiller/wiki - sim](https://huggingface.co/datasets/dleemiller/wiki - sim) dataset. This dataset provides diverse sentence pairs with semantic similarity scores, helping the model build a robust understanding of relationships between sentences.

• Classifier Dropout: a somewhat large classifier dropout of 0.3, to reduce overreliance on teacher scores.
• Objective: STS - B scores from cross - encoder/stsb - roberta - large.

Fine - Tuning

Fine - tuning was performed on the [sentence - transformers/stsb](https://huggingface.co/datasets/sentence - transformers/stsb) dataset.

Validation Results

The model achieved the following test set performance after fine - tuning:

• Pearson Correlation: 0.9256
• Spearman Correlation: 0.9215

Model Card

📄 License

This model is licensed under the MIT License.

🙏 Thank You

Thanks to the AnswerAI team for providing the ModernBERT models, and the Sentence Transformers team for their leadership in transformer encoder models.

📖 Citation

If you use this model in your research, please cite:

@misc{moderncestsb2025,
  author = {Miller, D. Lee},
  title = {ModernCE STS: An STS cross encoder model},
  year = {2025},
  publisher = {Hugging Face Hub},
  url = {https://huggingface.co/dleemiller/ModernCE-large-sts},
}