Swiss German Xlm Roberta Base

🚀 XLM - RoBERTa Base Continued Pre - trained on Swiss German

The XLM - RoBERTa base model continued pre - trained on Swiss German text data, enabling better performance on Swiss German language tasks.

🚀 Quick Start

The [xlm - roberta - base](https://huggingface.co/xlm - roberta - base) model ([Conneau et al., ACL 2020](https://aclanthology.org/2020.acl - main.747/)) is continued pre - trained on Swiss German text data.

Widget Example

I cha etz au Schwiizerdütsch. <mask> zäme! 😊

📚 Documentation

📦 Training Data

For continued pre - training, the following two datasets of written Swiss German were used:

1. SwissCrawl ([Linder et al., LREC 2020](https://aclanthology.org/2020.lrec - 1.329)), a collection of Swiss German web text (forum discussions, social media).
2. A custom dataset of Swiss German tweets

In addition, the model was trained on an equal amount of Standard German data. News articles retrieved from Swissdox@LiRI were used.

📄 License

Attribution - NonCommercial 4.0 International (CC BY - NC 4.0).

📚 Citation

@inproceedings{vamvas-etal-2024-modular,
      title={Modular Adaptation of Multilingual Encoders to Written Swiss German Dialect},
      author={Jannis Vamvas and No{\"e}mi Aepli and Rico Sennrich},
      booktitle={First Workshop on Modular and Open Multilingual NLP},
      year={2024},
}

📋 Information Table