Czert B Base Cased Long Zero Shot

🚀 CZERT

This repository stores the trained Czert-B-base-cased-long-zero-shot model for the paper Czert – Czech BERT-like Model for Language Representation. For more details, refer to the paper.

This is the long version of Czert-B-base-cased, created without any fine-tuning on long documents. Positional embeddings were generated by simply repeating the positional embeddings of the original Czert-B model. For tokenization, use BertTokenizer. It cannot be used with AutoTokenizer.

🚀 Quick Start

This README provides information about the CZERT models, including available models, usage scenarios, fine-tuning results, license, and citation details.

✨ Features

• Multiple Model Versions: Offers both pre-trained and fine-tuned models for various NLP tasks.
• Comprehensive Task Coverage: Evaluated on sentence, document, and token-level tasks such as sentiment classification, named entity recognition, etc.
• Comparative Results: Provides fine-tuning results compared with other models like mBERT, SlavicBERT, etc.

📦 Installation

The README does not provide specific installation steps. You can download the models from the provided links.

💻 Usage Examples

Sentence Level Tasks

We evaluate our model on two sentence level tasks:

• Sentiment Classification
• Semantic Text Similarity

Document Level Tasks

We evaluate our model on one document level task:

• Multi-label Document Classification

Token Level Tasks

We evaluate our model on three token level tasks:

• Named Entity Recognition
• Morphological Tagging
• Semantic Role Labelling

📚 Documentation

Available Models

You can download MLM & NSP only pretrained models: CZERT-A-v1 CZERT-B-v1

After additional experiments, we found that the tokenizers config was exported incorrectly. In Czert-B-v1, the tokenizer parameter "do_lower_case" was wrongly set to true. In Czert-A-v1, the parameter "strip_accents" was incorrectly set to true. Both mistakes are fixed in v2.

• CZERT-A-v2
• CZERT-B-v2

Or choose from one of Finetuned Models:

Downstream Tasks Fine-tuning Results

Sentiment Classification

Average F1 results for the Sentiment Classification task. For more information, see the paper.

Semantic Text Similarity

Comparison of Pearson correlation achieved using pre-trained CZERT-A, CZERT-B, mBERT, Pavlov and randomly initialised Albert on semantic text similarity. For more information see the paper.

Multi-label Document Classification

Comparison of F1 and AUROC score achieved using pre-trained CZERT-A, CZERT-B, mBERT, Pavlov and randomly initialised Albert on multi-label document classification. For more information see the paper.

Morphological Tagging

Comparison of F1 score achieved using pre-trained CZERT-A, CZERT-B, mBERT, Pavlov and randomly initialised Albert on morphological tagging task. For more information see the paper.

Semantic Role Labelling

SRL results – dep columns are evaluate with labelled F1 from CoNLL 2009 evaluation script, other columns are evaluated with span F1 score same as it was used for NER evaluation. For more information see the paper.

Named Entity Recognition

Comparison of f1 score achieved using pre-trained CZERT-A, CZERT-B, mBERT, Pavlov and randomly initialised Albert on named entity recognition task. For more information see the paper.

📄 License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. http://creativecommons.org/licenses/by-nc-sa/4.0/

How should I cite CZERT?

For now, please cite the Arxiv paper:

@article{sido2021czert,
      title={Czert -- Czech BERT-like Model for Language Representation}, 
      author={Jakub Sido and Ondřej Pražák and Pavel Přibáň and Jan Pašek and Michal Seják and Miloslav Konopík},
      year={2021},
      eprint={2103.13031},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      journal={arXiv preprint arXiv:2103.13031},
}