Electra Base Gc4 64k 500000 Cased Generator

🚀 GC4LM: A Colossal (Biased) language model for German

This repository offers a colossal (and biased) German language model trained on the newly released "German colossal, clean Common Crawl corpus" (GC4), with a total dataset size of approximately 844GB.

🚀 Quick Start

The model is trained on the GC4 corpus. You can access the relevant dataset details via the provided link: "German colossal, clean Common Crawl corpus".

✨ Features

• Large-scale Training: Trained on a ~844GB dataset, the "German colossal, clean Common Crawl corpus" (GC4).
• Research Focus: Aims to promote research on large pre - trained German language models, especially in identifying and preventing biases.

📦 Installation

No specific installation steps are provided in the original document.

📚 Documentation

Disclaimer

The language models presented and trained in this repository are for research only purposes. The GC4 corpus used for training contains crawled internet texts. As a result, the language models are highly biased, encoding stereotypical associations related to gender, race, ethnicity, and disability status.

Before using the released checkpoints, it is strongly recommended to read On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? by Emily M. Bender, Timnit Gebru, Angelina McMillan - Major and Shmargaret Shmitchell.

Discussion and Research

Please use the new GitHub Discussions feature to discuss or present further research questions. You can also use #gc4lm on Twitter 🐦.

📄 License

The project is licensed under the MIT license.

📦 Dataset

⚠️ Important Note

The language models in this repository are for research only. They are highly biased due to the nature of the training corpus.

💡 Usage Tip

Before using the released checkpoints, read On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? to understand the potential biases in language models.