Xlm Roberta Base Multilingual Text Genre Classifier

🚀 X-GENRE classifier - multilingual text genre classifier

This is a text classification model that can automatically identify text genres in multiple languages, offering high performance and reliability.

🚀 Quick Start

The X-GENRE classifier is a text classification model based on xlm-roberta-base and fine - tuned on a multilingual manually - annotated X - GENRE genre dataset. It can be used for automatic genre identification on any text in a language supported by the xlm - roberta - base.

The details on the model development, datasets, and the model's in - dataset, cross - dataset and multilingual performance are provided in the paper Automatic Genre Identification for Robust Enrichment of Massive Text Collections: Investigation of Classification Methods in the Era of Large Language Models (Kuzman et al., 2023).

The model can also be downloaded from the CLARIN.SI repository.

If you use the model, please cite the paper:

@article{kuzman2023automatic,
  title={Automatic Genre Identification for Robust Enrichment of Massive Text Collections: Investigation of Classification Methods in the Era of Large Language Models},
  author={Kuzman, Taja and Mozeti{\v{c}}, Igor and Ljube{\v{s}}i{\'c}, Nikola},
  journal={Machine Learning and Knowledge Extraction},
  volume={5},
  number={3},
  pages={1149--1175},
  year={2023},
  publisher={MDPI}
}

✨ Features

• Multilingual Support: Supports a wide range of languages including English, Slovene, French, etc.
• High Performance: Outperforms many other models in both English and multilingual test datasets.
• Free to Use: Can be used for thousands of different tasks at no cost.

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

from simpletransformers.classification import ClassificationModel
model_args= {
            "num_train_epochs": 15,
            "learning_rate": 1e-5,
            "max_seq_length": 512,
            "silent": True
            }
model = ClassificationModel(
    "xlmroberta", "classla/xlm-roberta-base-multilingual-text-genre-classifier", use_cuda=True,
    args=model_args
    
)
predictions, logit_output = model.predict(["How to create a good text classification model? First step is to prepare good data. Make sure not to skip the exploratory data analysis. Pre-process the text if necessary for the task. The next step is to perform hyperparameter search to find the optimum hyperparameters. After fine-tuning the model, you should look into the predictions and analyze the model's performance. You might want to perform the post-processing of data as well and keep only reliable predictions.", 
                                        "On our site, you can find a great genre identification model which you can use for thousands of different tasks. With our model, you can fastly and reliably obtain high-quality genre predictions and explore which genres exist in your corpora. Available for free!"]
                                        )
predictions
# Output: array([3, 8])

[model.config.id2label[i] for i in predictions]
# Output: ['Instruction', 'Promotion']

Advanced Usage

A use example for prediction on a dataset, using batch processing, is available via Google Collab.

📚 Documentation

AGILE - Automatic Genre Identification Benchmark

We set up a benchmark for evaluating the robustness of automatic genre identification models to test their usability for the automatic enrichment of large text collections with genre information. The benchmark comprises 11 European languages and two test datasets. You are welcome to submit your entry at the benchmark's GitHub repository.

The model outperforms all other technologies, including GPT models that are used in a zero - shot scenario.

Results on the English test dataset (EN - GINCO)

Results on the multilingual test dataset (X - GINCO)

(The multilingual test dataset is easier than the English one, as the vague label "Other" and instances that were predicted with a confidence score below 0.80 were not included in the test dataset.)

For language - specific results, see the AGILE benchmark.

Intended use and limitations

Usage

An example of preparing data for genre identification and post - processing of the results can be found here where we applied X - GENRE classifier to the English part of MaCoCu parallel corpora.

For reliable results, the genre classifier should be applied to documents of sufficient length (the rule of thumb is at least 75 words). It is advised that the predictions, predicted with confidence higher than 0.9, are not used. Furthermore, the label "Other" can be used as another indicator of low confidence of the predictions, as it often indicates that the text does not have enough features of any genre, and these predictions can be discarded as well.

After proposed post - processing (removal of low - confidence predictions, labels "Other" and in this specific case also label "Forum"), the performance on the MaCoCu data based on manual inspection reached macro and micro F1 of 0.92.

X - GENRE categories

List of labels

labels_list=['Other', 'Information/Explanation', 'News', 'Instruction', 'Opinion/Argumentation', 'Forum', 'Prose/Lyrical', 'Legal', 'Promotion'],

labels_map={'Other': 0, 'Information/Explanation': 1, 'News': 2, 'Instruction': 3, 'Opinion/Argumentation': 4, 'Forum': 5, 'Prose/Lyrical': 6, 'Legal': 7, 'Promotion': 8}

Description of labels

Fine - tuning hyperparameters

Fine - tuning was performed with simpletransformers. Beforehand, a brief hyperparameter optimization was performed and the presumed optimal hyperparameters are:

model_args= {
            "num_train_epochs": 15,
            "learning_rate": 1e-5,
            "max_seq_length": 512,
            "silent": True
            }

🔧 Technical Details

The details on the model development, the datasets and the model's in - dataset, cross - dataset and multilingual performance are provided in the paper Automatic Genre Identification for Robust Enrichment of Massive Text Collections: Investigation of Classification Methods in the Era of Large Language Models (Kuzman et al., 2023).

📄 License

This model is licensed under cc - by - sa - 4.0.