Modernbert Base Tr Uncased

🚀 Artiwise ModernBERT - Base Turkish Uncased

We present Artiwise ModernBERT for Turkish 🎉. It is a BERT model with a modernized architecture and an increased context size. (The context size of older BERT models is 512, while that of ModernBERT is 8192). This model is a Turkish adaptation of ModernBERT, fine - tuned from answerdotai/ModernBERT - base using only the Turkish part of CulturaX.

📦 Installation

Note: Torch version must be >= 2.6.0 and transformers version >= 4.50.0 for the model to function properly. Also Don't use the do_lower_case = True flag with the tokenizer. Instead, convert your text to lower case as follows:

text.replace("I", "ı").lower()

This is due to a known issue with the tokenizer.

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

# Load the model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("artiwise-ai/modernbert-base-tr-uncased")
model = AutoModelForMaskedLM.from_pretrained("artiwise-ai/modernbert-base-tr-uncased")

# Example sentence with masked token
text = "Türkiye'nin başkenti [MASK]'dır."
text.replace("I", "ı").lower()

# Tokenize and prepare input
inputs = tokenizer(text, return_tensors="pt")

# Get the position of the masked token
mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]

# Forward pass
with torch.no_grad():
    outputs = model(**inputs)

# Get the predictions for the masked token
logits = outputs.logits
mask_token_logits = logits[0, mask_token_index, :]
top_5_tokens = torch.topk(mask_token_logits, 5, dim=1).indices[0].tolist()

# Print the predictions
print(f"Original text: {text}")
print("Top 5 predictions for [MASK]:")
for token in top_5_tokens:
    print(f"- {tokenizer.decode([token])}")

📚 Documentation

Stats

Benchmark

The benchmark results below demonstrate that Artiwise ModernBERT consistently outperforms existing Turkish BERT variants across multiple domains and masking levels, highlighting its superior generalization capabilities.

Our experiments used three datasets: the [Turkish Biomedical Corpus](https://huggingface.co/datasets/hazal/Turkish - Biomedical - corpus - trM), the Turkish Product Reviews dataset, and the general - domain QA corpus turkish_v2.

📄 License

This project is under the MIT license.