Bert2d Cased Turkish 128K WWM NSW2

🚀 Bert2DModel

Bert2DModel is a novel adaptation of the classic BERT architecture, tailored for languages with complex word structures, such as Turkish. It offers a deeper understanding of grammar and meaning by leveraging a 2D embedding system.

🚀 Quick Start

Bert2DModel presents a fresh approach to the traditional BERT architecture. While regular BERT views a sentence as a simple linear sequence of words, some languages, like Turkish, have words with rich internal structures (prefixes, suffixes, etc.). Bert2D employs a "2D embedding" system. It not only considers a word's position in the sentence (the first dimension) but also the position of sub - pieces within the word (the second dimension). This enables a more profound understanding of grammar and meaning, especially for languages where words can take on numerous forms. The initial version is trained for Turkish!

You can find all the original [Bert2DModel] checkpoints under the yigitbekir collection.

⚠️ Important Note

Click on the [Bert2DModel] models in the right sidebar for more examples of how to apply [Bert2DModel] to different text and token classification tasks.

💻 Usage Examples

Basic Usage

The example below demonstrates how to use the fill - mask pipeline with Bert2DModel or load it directly with the [AutoModel] class.

from transformers import pipeline

# 1. Define your model repository ID
repo_id = "yigitbekir/Bert2D-cased-Turkish-128K-WWM-NSW2"

# 2. Create the pipeline for the "fill-mask" task
# The model_kwargs dictionary passes arguments to the underlying model loading function.
fill_masker = pipeline(
    "fill-mask",
    model=repo_id,
    use_fast=True,
    trust_remote_code=True
)

# 3. Prepare the input and get predictions
masked_sentence = "Adamın mesleği [MASK] midir acaba?"
predictions = fill_masker(masked_sentence)

# 4. Print the results in a user-friendly format
print(f"Predictions for: '{masked_sentence}'")
for prediction in predictions:
    print(f"  Sequence: {prediction['sequence']}")
    print(f"  Token: {prediction['token_str']}")
    print(f"  Score: {prediction['score']:.4f}")
    print("-" * 20)

# Expected output:
# Sequence: Adamın mesleği mühendis midir acaba?
# Score: 0.2393
# --------------------
# Sequence: Adamın mesleği doktor midir acaba?
# Score: 0.1698
# --------------------

Advanced Usage

from transformers import AutoTokenizer, AutoModel

# Load the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("yigitbekir/Bert2D-cased-Turkish-128K-WWM-NSW2", trust_remote_code=True)
model = AutoModel.from_pretrained("yigitbekir/Bert2D-cased-Turkish-128K-WWM-NSW2", trust_remote_code=True)

# Example text
text = "Türkiye'nin başkenti Ankara'dır."
inputs = tokenizer(text, return_tensors="pt")

# Get model outputs
outputs = model(**inputs)
last_hidden_states = outputs.last_hidden_state

🔧 Technical Details

Configuration is Key

Bert2D introduces new configuration parameters that are not present in a standard BERT model. You must use the Bert2DConfig and be mindful of these settings when training or fine - tuning. Failing to do so will lead to unexpected behavior. The two key new parameters are max_word_position_embeddings and max_intermediate_subword_position_embeddings.

from transformers import AutoConfig

# Load the custom config from a pretrained model
config = AutoConfig.from_pretrained("yigitbekir/Bert2D-cased-Turkish-128K-WWM-NSW2", trust_remote_code=True)
  
# Access new parameters
print(f"Max Word Positions: {config.max_word_position_embeddings}")
# Expected output: Max Word Positions: 512
  
print(f"Intermediate Subword Position: {config.max_intermediate_subword_position_embeddings}")
# Expected output: Intermediate Subword Position: 2

📄 License

This project is licensed under the Apache 2.0 license.