Elhberteu

🚀 ElhBERTeu

This is a BERT model for Basque introduced in BasqueGLUE: A Natural Language Understanding Benchmark for Basque. It aims to enhance natural language understanding capabilities for the Basque language.

✨ Features

• Diverse Corpora Utilization: To train ElhBERTeu, we gathered different corpora sources from multiple domains. These include updated (2021) national and local news sources, Basque Wikipedia, novel news sources, and texts from various other domains such as science (both academic and divulgative), literature, or subtitles.
• Model Specifications: ElhBERTeu is a base, cased monolingual BERT model for Basque. It has a vocab size of 50K and a total of 124M parameters.
• Medium - size Variant: There is a medium - size model available here: [ElhBERTeu - medium](https://huggingface.co/orai - nlp/ElhBERTeu - medium).

📚 Documentation

Corpora Details

To train ElhBERTeu, we collected different corpora sources from several domains. Texts from news sources were oversampled (duplicated) as done during the training of BERTeus. In total 575M tokens were used for pre - training ElhBERTeu. The following table shows more details about the corpora used and their sizes:

Model Training

ElhBERTeu was trained following the design decisions for [BERTeus](https://huggingface.co/ixa - ehu/berteus - base - cased). The tokenizer and the hyper - parameter settings remained the same (batch_size = 256), with the only difference being that the full pre - training of the model (1M steps) was performed with a sequence length of 512 on a v3 - 8 TPU.

Model Evaluation

The model has been evaluated on the recently created BasqueGLUE NLU benchmark:

📄 License

This project is licensed under the CC BY 4.0 license.

📖 Citation

If you use this model, please cite the following paper:

• G. Urbizu, I. San Vicente, X. Saralegi, R. Agerri, A. Soroa. BasqueGLUE: A Natural Language Understanding Benchmark for Basque. In proceedings of the 13th Language Resources and Evaluation Conference (LREC 2022). June 2022. Marseille, France

@InProceedings{urbizu2022basqueglue,
  author    = {Urbizu, Gorka  and  San Vicente, Iñaki  and  Saralegi, Xabier  and  Agerri, Rodrigo  and  Soroa, Aitor},
  title     = {BasqueGLUE: A Natural Language Understanding Benchmark for Basque},
  booktitle      = {Proceedings of the Language Resources and Evaluation Conference},
  month          = {June},
  year           = {2022},
  address        = {Marseille, France},
  publisher      = {European Language Resources Association},
  pages     = {1603--1612},
  abstract  = {Natural Language Understanding (NLU) technology has improved significantly over the last few years and multitask benchmarks such as GLUE are key to evaluate this improvement in a robust and general way. These benchmarks take into account a wide and diverse set of NLU tasks that require some form of language understanding, beyond the detection of superficial, textual clues. However, they are costly to develop and language-dependent, and therefore they are only available for a small number of languages. In this paper, we present BasqueGLUE, the first NLU benchmark for Basque, a less-resourced language, which has been elaborated from previously existing datasets and following similar criteria to those used for the construction of GLUE and SuperGLUE. We also report the evaluation of two state-of-the-art language models for Basque on BasqueGLUE, thus providing a strong baseline to compare upon. BasqueGLUE is freely available under an open license.},
  url       = {https://aclanthology.org/2022.lrec-1.172}
}