Gottbert Base Last

🚀 GottBERT: A pure German language model

GottBERT is the first German-only RoBERTa model, pre-trained on the German portion of the first released OSCAR dataset. It aims to enhance natural language processing (NLP) performance for German across various tasks, such as Named Entity Recognition (NER), text classification, and natural language inference (NLI). There are two versions of GottBERT: a base model and a large model, both tailored for German-language tasks.

This model was introduced in GottBERT: a pure German Language Model.

📚 Documentation

Pretraining Details

• Corpus: German portion of the OSCAR dataset (Common Crawl).
• Data Size:
  • Unfiltered: 145GB (~459 million documents)
  • Filtered: 121GB (~382 million documents)
• Preprocessing: Filtering involved correcting encoding errors (e.g., erroneous umlauts), removing spam and non-German documents using language detection and syntactic filtering.

Filtering Metrics

• Stopword Ratio: Detects spam and meaningless content.
• Punctuation Ratio: Detects abnormal punctuation patterns.
• Upper Token Ratio: Identifies documents with excessive uppercase tokens (often noisy content).

Training Configuration

• Framework: Fairseq
• Hardware:
  • Base Model: 256 TPUv3 pod/128 TPUv4 pod
  • Large Model: 128 TPUv4 pod
• Training Time:
  • Base Model: 1.2 days
  • Large Model: 5.7 days
• Batch Size: 8k tokens
• Learning Rate:
  • Base: Peak LR = 0.0004
  • Large: Peak LR = 0.00015
• Training Iterations: 100k steps with a 10k warm-up phase

Evaluation and Results

GottBERT was evaluated on various downstream tasks:

• NER: CoNLL 2003, GermEval 2014
• Text Classification: GermEval 2018 (coarse & fine), 10kGNAD
• NLI: German subset of XNLI

Metrics:

• NER and Text Classification: F1 Score
• NLI: Accuracy

Details:

• Bold values indicate the best performing model within one architecture (base, large), underlined values the second best.

Model Architecture

• Base Model: 12 layers, 125M parameters, 52k token vocabulary.
• Large Model: 24 layers, 355M parameters, 52k token vocabulary.

Tokenizer

• Type: GPT-2 Byte-Pair Encoding (BPE)
• Vocabulary Size: 52k subword tokens
• Trained on: 40GB subsample of the unfiltered German OSCAR corpus.

Limitations

• Filtered vs Unfiltered Data: Filtered data shows minor improvements, but not significant enough to justify filtering in all cases.
• Computation Limitations: Fixed memory allocation on TPUs requires processing data as a single stream, unlike GPU training which preserves document boundaries. Due to framework limitations, training was performed in 32-bit mode, increasing memory usage.

📄 License

The model is released under the MIT license.

🔗 Fairseq Checkpoints

Get the fairseq checkpoints here.

📖 Citations

If you use GottBERT in your research, please cite the following paper:

@inproceedings{scheible-etal-2024-gottbert,
    title = "{G}ott{BERT}: a pure {G}erman Language Model",
    author = "Scheible, Raphael  and
      Frei, Johann  and
      Thomczyk, Fabian  and
      He, Henry  and
      Tippmann, Patric  and
      Knaus, Jochen  and
      Jaravine, Victor  and
      Kramer, Frank  and
      Boeker, Martin",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-main.1183",
    pages = "21237--21250",
}