roberta-base-wechsel-ukrainian Open Source NLP Model - Excellent Performance in Multiple Ukrainian Tasks

Roberta Base Wechsel Ukrainian

Developed by benjamin

This model is a Ukrainian version of roberta-base migrated via the WECHSEL method, demonstrating excellent performance across multiple Ukrainian NLP tasks.

Large Language Model

Transformers

OtherOpen Source License:MIT #Ukrainian NLP #WECHSEL transfer learning #High-accuracy part-of-speech tagging

Downloads 16

Release Time : 4/3/2022

Model Overview

A Ukrainian pre-trained model transferred from the English roberta-base model using WECHSEL cross-lingual transfer technology, suitable for Ukrainian natural language processing tasks.

Model Features

Efficient cross-lingual transfer

Utilizes the WECHSEL method for efficient parameter transfer from English to Ukrainian, achieving better results compared to training from scratch.

Excellent Ukrainian language processing capability

Outperforms similar models in Ukrainian NER and part-of-speech tagging tasks.

Dual evaluation validation

Evaluated on multiple datasets including lang-uk's ner-uk project and the Ukrainian portion of WikiANN.

Model Capabilities

Ukrainian text understanding

Named entity recognition

Part-of-speech tagging

Use Cases

Natural language processing

Ukrainian text entity recognition

Identifies entities such as person names, locations, and organizations in Ukrainian text.

Achieved 91.24 F1 score on the lang-uk NER test set.

Ukrainian part-of-speech tagging

Tags parts of speech for words in Ukrainian text.

Achieved 98.74% accuracy on the UD Ukrainian IU corpus.

🚀 roberta-base-wechsel-ukrainian

The roberta-base model transferred to Ukrainian using the method from the NAACL2022 paper WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

🚀 Quick Start

The roberta-base model has been transferred to Ukrainian using the method from the NAACL2022 paper WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

📚 Documentation

Evaluation

Evaluation was done on lang-uk's ner-uk project, the Ukrainian portion of WikiANN and the Ukrainian IU corpus from the Universal Dependencies project. Evaluation results are the mean of 5 runs with different seeds.

Validation Results

Property	lang-uk NER (Micro F1)	WikiANN (Micro F1)	UD Ukrainian IU POS (Accuracy)
roberta-base-wechsel-ukrainian	88.06 (0.50)	92.96 (0.08)	98.70 (0.05)
roberta-large-wechsel-ukrainian	89.27 (0.53)	93.22 (0.15)	98.86 (0.03)
roberta-base-scratch-ukrainian*	85.49 (0.88)	91.91 (0.08)	98.49 (0.04)
roberta-large-scratch-ukrainian*	86.54 (0.70)	92.39 (0.16)	98.65 (0.09)
dbmdz/electra-base-ukrainian-cased-discriminator	87.49 (0.52)	93.20 (0.16)	98.60 (0.03)
xlm-roberta-base	86.68 (0.44)	92.41 (0.13)	98.53 (0.02)
xlm-roberta-large	86.64 (1.61)	93.01 (0.13)	98.71 (0.04)

Test Results

Property	lang-uk NER (Micro F1)	WikiANN (Micro F1)	UD Ukrainian IU POS (Accuracy)
roberta-base-wechsel-ukrainian	90.81 (1.51)	92.98 (0.12)	98.57 (0.03)
roberta-large-wechsel-ukrainian	91.24 (1.16)	93.22 (0.17)	98.74 (0.06)
roberta-base-scratch-ukrainian*	89.57 (1.01)	92.05 (0.09)	98.31 (0.08)
roberta-large-scratch-ukrainian*	89.96 (0.89)	92.49 (0.15)	98.52 (0.04)
dbmdz/electra-base-ukrainian-cased-discriminator	90.43 (1.29)	92.99 (0.11)	98.59 (0.06)
xlm-roberta-base	90.86 (0.81)	92.27 (0.09)	98.45 (0.07)
xlm-roberta-large	90.16 (2.98)	92.92 (0.19)	98.71 (0.04)

*trained using the same exact training setup as the wechsel-* models, but without parameter transfer from WECHSEL.

📄 License

MIT

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご