Mk Roberta Base

🚀 MK-RoBERTa base model

A pre - trained model on the Macedonian language using a masked language modeling (MLM) objective.

🚀 Quick Start

The MK - RoBERTa base model is pre - trained on Macedonian language data with a masked language modeling (MLM) objective. It was introduced in a paper and first released in a specific repository. This model is case - sensitive, distinguishing between "скопје" and "Скопје".

✨ Features

• Self - supervised pre - training: RoBERTa is pre - trained on a large corpus of Macedonian data in a self - supervised way. It uses an automatic process to generate inputs and labels from raw texts without human labeling, which allows it to utilize a large amount of publicly available data.
• Masked language modeling (MLM): The model randomly masks 15% of the words in the input sentence and then predicts the masked words. This enables the model to learn a bidirectional representation of the sentence, different from traditional RNNs and autoregressive models like GPT.
• Feature extraction for downstream tasks: The model can learn an inner representation of the Macedonian language, which can be used to extract features for downstream tasks such as sequence classification, token classification, or question answering.

📚 Documentation

Model description

RoBERTa is a transformers model pre - trained on a large corpus of Macedonian data in a self - supervised fashion. It uses an automatic process to generate inputs and labels from raw texts without any human labeling. This allows it to make use of a vast amount of publicly available data.

Specifically, it is pre - trained with the Masked language modeling (MLM) objective. Given a sentence, the model randomly masks 15% of the words in the input, passes the masked sentence through the model, and then predicts the masked words. This is different from traditional recurrent neural networks (RNNs) that process words sequentially and autoregressive models like GPT that mask future tokens internally. This mechanism enables the model to learn a bidirectional representation of the sentence.

In this way, the model learns an internal representation of the Macedonian language, which can be used to extract features for downstream tasks. For example, if you have a dataset of labeled sentences, you can train a standard classifier using the features generated by the RoBERTa model as inputs.

Intended uses & limitations

You can use the raw model for masked language modeling, but it is mainly intended to be fine - tuned on downstream tasks. Check the model hub for fine - tuned versions of tasks that interest you.

Note that this model is primarily designed to be fine - tuned on tasks that use the whole sentence (possibly masked) for decision - making, such as sequence classification, token classification, or question answering. For tasks like text generation, you should consider models like GPT2.

💻 Usage Examples

Basic Usage

You can use this model directly with a pipeline for masked language modeling:

from transformers import pipeline
unmasker = pipeline('fill - mask', model='macedonizer/mk - roberta - base')
unmasker("Скопје е <mask> град на Македонија.")

[{'sequence': 'Скопје е главен град на Македонија.',
    'score': 0.5900368094444275,
    'token': 2782,
    'token_str': ' главен'},
  {'sequence': 'Скопје е главниот град на Македонија.',
   'score': 0.1789761781692505,
   'token': 3177,
   'token_str': ' главниот'},
  {'sequence': 'Скопје е административен град на Македонија.',
   'score': 0.01679774932563305,
   'token': 9563,
   'token_str': ' административен'},
  {'sequence': 'Скопје е мал град на Македонија.',
   'score': 0.016263898462057114,
   'token': 2473,
   'token_str': ' мал'},
  {'sequence': 'Скопје е најголемиот град на Македонија.',
   'score': 0.01312252413481474,
   'token': 4271,
   'token_str': ' најголемиот'}]

Advanced Usage

Here is how to use this model to get the features of a given text in PyTorch:

from transformers import RobertaTokenizer, RobertaModel
tokenizer = RobertaTokenizer.from_pretrained('macedonizer/mk - roberta - base')
model = RobertaModel.from_pretrained('macedonizer/mk - roberta - base')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

📄 License

This model is licensed under the Apache 2.0 license.