Morphbert Large Morpheme Segmentation Ru

🚀 MorphBERT-Large: Russian Morpheme Segmentation

This repository houses the CrabInHoney/morphbert-large-morpheme-segmentation-ru model, a large transformer-based system fine-tuned for morpheme segmentation of Russian words. The model classifies each character of a given word into one of 25 morpheme categories: ['END', 'END1', 'HYPH', 'HYPH1', 'LINK', 'LINK1', 'LINK2', 'LINK3', 'POSTFIX', 'PREF', 'PREF1', 'PREF2', 'ROOT', 'ROOT1', 'ROOT2', 'ROOT3', 'ROOT4', 'ROOT5', 'SUFF', 'SUFF1', 'SUFF2', 'SUFF3', 'SUFF4', 'SUFF5', 'SUFF6']

✨ Features

• Task: Morpheme Segmentation (Token Classification at Character Level)
• Language: Russian (ru)
• Architecture: Transformer (BERT base -like)
• Labels: ['END', 'END1', 'HYPH', 'HYPH1', 'LINK', 'LINK1', 'LINK2', 'LINK3', 'POSTFIX', 'PREF', 'PREF1', 'PREF2', 'ROOT', 'ROOT1', 'ROOT2', 'ROOT3', 'ROOT4', 'ROOT5', 'SUFF', 'SUFF1', 'SUFF2', 'SUFF3', 'SUFF4', 'SUFF5', 'SUFF6']

📦 Installation

The model can be easily used with the Hugging Face transformers library. You can install it using the following command:

pip install transformers torch

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

model_name = "CrabInHoney/morphbert-large-morpheme-segmentation-ru"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)
model.eval()

def analyze(word):
    tokens = list(word)
    encoded = tokenizer(tokens, is_split_into_words=True, return_tensors="pt", truncation=True, max_length=34)
    with torch.no_grad():
        logits = model(**encoded).logits
        predictions = logits.argmax(dim=-1)[0]
    
    word_ids = encoded.word_ids()
    output = []
    current_label = None
    current_chunk = []

    for i, word_idx in enumerate(word_ids):
        if word_idx is not None and word_idx < len(tokens):
            label_id = predictions[i].item()
            label = model.config.id2label[label_id]  
            token = tokens[word_idx]
            
            if label == current_label:
                current_chunk.append(token)
            else:
                if current_chunk:
                    chunk_str = "".join(current_chunk)
                    output.append(f"{chunk_str}:{current_label}")
                current_chunk = [token]
                current_label = label
    if current_chunk:
        chunk_str = "".join(current_chunk)
        output.append(f"{chunk_str}:{current_label}")

    return " / ".join(output)

# Examples
for word in ["масляный", "предчувствий", "тарковский", "кот", "подгон", "сине-белый", "шторы", "абажур", "дедлайн", "веб-сайт", "адаптированная", "формообразующий"]:
    print(f"{word} → {analyze(word)}")

Example Predictions

масляный → масл:ROOT / ян:SUFF / ый:END
предчувствий → пред:PREF / чу:ROOT / в:SUFF / ств:SUFF1 / ий:END
тарковский → тарк:ROOT / ов:SUFF / ск:SUFF1 / ий:END
кот → кот:ROOT
подгон → под:PREF / гон:ROOT
сине-белый → син:ROOT / е:LINK / -:HYPH / бел:ROOT1 / ый:END
шторы → штор:ROOT / ы:END
абажур → абажур:ROOT
дедлайн → дедлайн:ROOT
веб-сайт → веб:ROOT / -:HYPH / сайт:ROOT1
адаптированная → адапт:ROOT / ир:SUFF / ова:SUFF1 / нн:SUFF2 / ая:END
формообразующий → форм:ROOT / о:LINK / образу:ROOT1 / ющ:SUFF / ий:END

📚 Documentation

Model Description

morphbert-large-morpheme-segmentation-ru uses the powerful transformer architecture, aimed at more accurate prediction of morphological analysis at the character level. Due to its large size, the model demonstrates greater accuracy in determining the constituent morphemes in Russian words compared to the small version (CrabInHoney/morphbert-tiny-morpheme-segmentation-ru).

The model was obtained by learning from scratch, and its architecture is comparable in complexity to bert-base.

Model Size & Specifications

Performance

The model achieves an approximate character-level accuracy of 0.99 on its evaluation dataset.

Limitations

• Performance may vary on out-of-vocabulary words, neologisms, or highly complex morphological structures not sufficiently represented in the training data.
• The model operates strictly at the character level; it does not incorporate broader lexical or syntactic context.
• Ambiguous cases in morpheme boundaries might be resolved based on patterns learned during training, which may not always align with linguistic conventions in edge cases.

📄 License

This project is licensed under the Apache-2.0 license.