Slovakbert

🚀 SlovakBERT (base-sized model)

SlovakBERT is a pre - trained model on the Slovak language using a masked language modeling (MLM) objective. It is case - sensitive and can distinguish between different cases of words.

🚀 Quick Start

SlovakBERT is a pre - trained model for the Slovak language with a masked language modeling (MLM) objective. It is case - sensitive. You can use the raw model for masked language modeling, but it's mainly intended for fine - tuning on downstream tasks.

⚠️ Important Note

The model was not trained on the “ and ” (direct quote) character. So, before tokenizing the text, it is advised to replace all “ and ” (direct quote marks) with a single "(double quote marks).

✨ Features

• Case - sensitive: It makes a difference between slovensko and Slovensko.
• Suitable for masked language modeling and fine - tuning on downstream tasks.

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

You can use this model directly with a pipeline for masked language modeling:

from transformers import pipeline
unmasker = pipeline('fill - mask', model='gerulata/slovakbert')
unmasker("Deti sa <mask> na ihrisku.")

[{'sequence': 'Deti sa hrali na ihrisku.',
  'score': 0.6355380415916443,
  'token': 5949,
  'token_str': ' hrali'},
 {'sequence': 'Deti sa hrajú na ihrisku.',
  'score': 0.14731724560260773,
  'token': 9081,
  'token_str': ' hrajú'},
 {'sequence': 'Deti sa zahrali na ihrisku.',
  'score': 0.05016357824206352,
  'token': 32553,
  'token_str': ' zahrali'},
 {'sequence': 'Deti sa stretli na ihrisku.',
  'score': 0.041727423667907715,
  'token': 5964,
  'token_str': ' stretli'},
 {'sequence': 'Deti sa učia na ihrisku.',
  'score': 0.01886524073779583,
  'token': 18099,
  'token_str': ' učia'}]

Advanced Usage

Here is how to use this model to get the features of a given text in PyTorch:

from transformers import RobertaTokenizer, RobertaModel
tokenizer = RobertaTokenizer.from_pretrained('gerulata/slovakbert')
model = RobertaModel.from_pretrained('gerulata/slovakbert')
text = "Text ktorý sa má embedovať."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

And in TensorFlow:

from transformers import RobertaTokenizer, TFRobertaModel
tokenizer = RobertaTokenizer.from_pretrained('gerulata/slovakbert')
model = TFRobertaModel.from_pretrained('gerulata/slovakbert')
text = "Text ktorý sa má embedovať."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)

Or extract information from the model like this:

from transformers import pipeline
unmasker = pipeline('fill - mask', model='gerulata/slovakbert')
unmasker("Slovenské národne povstanie sa uskutočnilo v roku <mask>.")

[{'sequence': 'Slovenske narodne povstanie sa uskutočnilo v roku 1944.',
  'score': 0.7383289933204651,
  'token': 16621,
  'token_str': ' 1944'},...]

📚 Documentation

Training data

The SlovakBERT model was pretrained on these datasets:

• Wikipedia (326MB of text),
• OpenSubtitles (415MB of text),
• Oscar (4.6GB of text),
• Gerulata WebCrawl (12.7GB of text),
• Gerulata Monitoring (214 MB of text),
• blbec.online (4.5GB of text)

The text was then processed with the following steps:

• URL and email addresses were replaced with special tokens ("url", "email").
• Elongated interpunction was reduced (e.g. -- to -).
• Markdown syntax was deleted.
• All text content in braces f.g was eliminated to reduce the amount of markup and programming language text.

We segmented the resulting corpus into sentences and removed duplicates to get 181.6M unique sentences. In total, the final corpus has 19.35GB of text.

Pretraining

The model was trained in fairseq on 4 x Nvidia A100 GPUs for 300K steps with a batch size of 512 and a sequence length of 512. The optimizer used is Adam with a learning rate of 5e - 4, \(\beta_{1}=0.9\), \(\beta_{2}=0.98\) and \(\epsilon = 1e - 6\), a weight decay of 0.01, dropout rate 0.1, learning rate warmup for 10k steps and linear decay of the learning rate after. We used 16 - bit float precision.

🔧 Technical Details

The model is trained using the masked language modeling (MLM) objective. It is case - sensitive, which means it can distinguish between different cases of words. The training process involves multiple steps of data pre - processing and specific hyperparameters for training on GPUs.

📄 License

The model is released under the MIT license.

About us

Gerulata Technologies is a tech company on a mission to provide tools for fighting disinformation and hostile propaganda.

At Gerulata, we focus on providing state - of - the - art AI - powered tools that empower human analysts and provide them with the ability to make informed decisions.

Our tools allow for the monitoring and analysis of online activity, as well as the detection and tracking of disinformation and hostile propaganda campaigns. With our products, our clients are better equipped to identify and respond to threats in real - time.

BibTeX entry and citation info

If you find our resource or paper is useful, please consider including the following citation in your paper.

• https://arxiv.org/abs/2109.15254

@misc{pikuliak2021slovakbert,
      title={SlovakBERT: Slovak Masked Language Model}, 
      author={Matúš Pikuliak and Štefan Grivalský and Martin Konôpka and Miroslav Blšták and Martin Tamajka and Viktor Bachratý and Marián Šimko and Pavol Balážik and Michal Trnka and Filip Uhlárik},
      year={2021},
      eprint={2109.15254},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}