tags:

• cs
• fill-mask

CZERT

This repository keeps trained Czert-B model for the paper Czert – Czech BERT-like Model for Language Representation For more information, see the paper

Available Models

You can download MLM & NSP only pretrained models CZERT-A-v1 CZERT-B-v1

After some additional experiments, we found out that the tokenizers config was exported wrongly. In Czert-B-v1, the tokenizer parameter "do_lower_case" was wrongly set to true. In Czert-A-v1 the parameter "strip_accents" was incorrectly set to true.

Both mistakes are repaired in v2. CZERT-A-v2 CZERT-B-v2

or choose from one of Finetuned Models

How to Use CZERT?

Sentence Level Tasks

We evaluate our model on two sentence level tasks:

• Sentiment Classification,
• Semantic Text Similarity.

\t

Document Level Tasks

We evaluate our model on one document level task

• Multi-label Document Classification.

Token Level Tasks

We evaluate our model on three token level tasks:

• Named Entity Recognition,
• Morphological Tagging,
• Semantic Role Labelling.

Downstream Tasks Fine-tuning Results

Sentiment Classification

Average F1 results for the Sentiment Classification task. For more information, see the paper.

Semantic Text Similarity

Comparison of Pearson correlation achieved using pre-trained CZERT-A, CZERT-B, mBERT, Pavlov and randomly initialised Albert on semantic text similarity. For more information see the paper.

Multi-label Document Classification

Comparison of F1 and AUROC score achieved using pre-trained CZERT-A, CZERT-B, mBERT, Pavlov and randomly initialised Albert on multi-label document classification. For more information see the paper.

Morphological Tagging

Comparison of F1 score achieved using pre-trained CZERT-A, CZERT-B, mBERT, Pavlov and randomly initialised Albert on morphological tagging task. For more information see the paper.

Semantic Role Labelling

Named Entity Recognition

Comparison of f1 score achieved using pre-trained CZERT-A, CZERT-B, mBERT, Pavlov and randomly initialised Albert on named entity recognition task. For more information see the paper.

Licence

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. http://creativecommons.org/licenses/by-nc-sa/4.0/

How should I cite CZERT?

For now, please cite the Arxiv paper:

@article{sido2021czert,
      title={Czert -- Czech BERT-like Model for Language Representation}, 
      author={Jakub Sido and Ondřej Pražák and Pavel Přibáň and Jan Pašek and Michal Seják and Miloslav Konopík},
      year={2021},
      eprint={2103.13031},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      journal={arXiv preprint arXiv:2103.13031},
}