BERTu Open-Source Maltese Language Model - Supports Multiple Types of Natural Language Processing and Boosts Low-Resource Language Applications

Bertu

Developed by MLRS

BERTu is a monolingual Maltese model based on the BERT architecture, specifically designed for low-resource languages, supporting various natural language processing tasks.

Large Language Model

Transformers

Other#Maltese NLP #Dependency Parsing #Low-resource Language Optimization

Downloads 4,486

Release Time : 4/14/2022

Model Overview

BERTu is a monolingual Maltese model pre-trained from scratch on Korpus Malti v4.0, based on the BERT (base) architecture. It supports multiple natural language processing tasks including dependency parsing, part-of-speech tagging, named entity recognition, and sentiment analysis.

Model Features

Low-resource Language Optimization

Designed specifically for low-resource languages like Maltese, it enhances model performance through high-quality corpus pre-training.

Multi-task Support

Supports various natural language processing tasks, including dependency parsing, part-of-speech tagging, named entity recognition, and sentiment analysis.

High Performance

Excels in multiple tasks, achieving a UAS of 92.31 in dependency parsing and an accuracy of over 98.5% in part-of-speech tagging.

Model Capabilities

Dependency parsing

Part-of-speech tagging

Named entity recognition

Sentiment analysis

Use Cases

Natural Language Processing

Dependency Parsing

Analyzes the syntactic structure of Maltese sentences, identifying dependency relationships between words.

Unlabeled Attachment Score (UAS) 92.31, Labeled Attachment Score (LAS) 88.14.

Part-of-Speech Tagging

Tags each word in Maltese text with its part of speech.

Universal POS tagging accuracy (UPOS) 98.58, language-specific POS tagging accuracy (XPOS) 98.54.

Named Entity Recognition

Identifies named entities (e.g., person names, locations, organizations) in Maltese text.

Span-based F1 score 86.77.

Sentiment Analysis

Analyzes the sentiment polarity (e.g., positive, negative, neutral) of Maltese text.

Macro-average F1 score 78.96.

🚀 BERTu

A Maltese monolingual model pre-trained from scratch on the Korpus Malti v4.0 using the BERT (base) architecture, offering high performance in multiple NLP tasks.

🚀 Quick Start

This section is not provided in the original document, so it is skipped.

✨ Features

Multilingual Support: Specifically designed for the Maltese language, trained on the Korpus Malti v4.0.
Multiple Task Capabilities: Demonstrates excellent performance in tasks such as dependency parsing, part - of - speech tagging, named entity recognition, and sentiment analysis.

📦 Installation

This section is not provided in the original document, so it is skipped.

💻 Usage Examples

This section is not provided in the original document, so it is skipped.

📚 Documentation

Model Information

Property	Details
Model Type	BERTu, a Maltese monolingual model
Training Data	MLRS/korpus_malti v4.0

Evaluation Results

Dependency Parsing

Dataset: Maltese Universal Dependencies Treebank (MUDT)
Metrics:
- Unlabelled Attachment Score (UAS): 92.31
- Labelled Attachment Score (LAS): 88.14

Part - of - Speech Tagging

Dataset: MLRS POS dataset
Metrics:
- UPOS Accuracy: 98.58
- XPOS Accuracy: 98.54

Named Entity Recognition

Dataset: WikiAnn (Maltese)
Metrics:
- Span - based F1: 86.77

Sentiment Analysis

Dataset: Maltese Sentiment Analysis Dataset
Metrics:
- Macro - averaged F1: 78.96

🔧 Technical Details

This section is not provided in the original document, so it is skipped.

📄 License

This work is licensed under a [Creative Commons Attribution - NonCommercial - ShareAlike 4.0 International License][cc - by - nc - sa]. Permissions beyond the scope of this license may be available at https://mlrs.research.um.edu.mt/.

[![CC BY - NC - SA 4.0][cc - by - nc - sa - image]][cc - by - nc - sa]

[cc - by - nc - sa]: http://creativecommons.org/licenses/by - nc - sa/4.0/ [cc - by - nc - sa - image]: https://licensebuttons.net/l/by - nc - sa/4.0/88x31.png

📚 Citation

This work was first presented in [Pre - training Data Quality and Quantity for a Low - Resource Language: New Corpus and BERT Models for Maltese](https://aclanthology.org/2022.deeplo - 1.10/). Cite it as follows:

@inproceedings{BERTu,
    title = "Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and {BERT} Models for {M}altese",
    author = "Micallef, Kurt  and
              Gatt, Albert  and
              Tanti, Marc  and
              van der Plas, Lonneke  and
              Borg, Claudia",
    booktitle = "Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing",
    month = jul,
    year = "2022",
    address = "Hybrid",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.deeplo-1.10",
    doi = "10.18653/v1/2022.deeplo-1.10",
    pages = "90--101",
}

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご