🚀 GerPT2
German large and small versions of GPT2, offering high - performance language processing capabilities.
🚀 Quick Start
GerPT2 provides German large and small versions of GPT2. You can access them through the following links:
- https://huggingface.co/benjamin/gerpt2
- https://huggingface.co/benjamin/gerpt2-large
For considerations on limitations and bias, refer to the GPT2 model card. For details on GPT2, see the GPT2 documentation.
✨ Features
The author evaluated both GerPT2 - large and dbmdz/german - gpt2 on the CC - 100 dataset and the German Wikipedia. The results are as follows:
Property |
CC - 100 (PPL) |
Wikipedia (PPL) |
dbmdz/german - gpt2 |
49.47 |
62.92 |
GerPT2 |
24.78 |
35.33 |
GerPT2 - large |
16.08 |
23.26 |
You can find the evaluation code in the evaluate.py
script in the GerPT2 Github repository.
💻 Usage Examples
Basic Usage
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
tokenizer = AutoTokenizer.from_pretrained("benjamin/gerpt2-large")
model = AutoModelForCausalLM.from_pretrained("benjamin/gerpt2-large")
prompt = "<your prompt>"
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
print(pipe(prompt)[0]["generated_text"])
Advanced Usage
import torch
output = model.generate(
torch.tensor(
[tokenizer.eos_token_id] + tokenizer.encode(prompt)
).unsqueeze(0),
do_sample=True,
bad_words_ids=[[0]],
max_length=max_length,
)[0]
print(tokenizer.decode(output))
🔧 Technical Details
Training Details
GerPT2 - large is trained on the entire German data from the CC - 100 Corpus, and its weights are initialized from the English GPT2 model.
GerPT2 - large was trained with the following settings:
- A batch size of 256
- OneCycle learning rate with a maximum of 5e - 3
- AdamW with a weight decay of 0.01
- For 2 epochs
The training took roughly 12 days on 8 TPUv3 cores.
To train GerPT2 - large, follow these steps. The scripts are located in the Github repository:
0. Download and unzip training data from http://data.statmt.org/cc-100/.
- Train a tokenizer using
prepare/train_tokenizer.py
. The author used a random subset of 5% of the CC - 100 data as training data for the tokenizer.
- (Optionally) Generate a German input embedding matrix with
prepare/generate_aligned_wte.py
. This script uses a neat trick to semantically map tokens from the English tokenizer to tokens from the German tokenizer using aligned word embeddings. For example:
ĠMinde -> Ġleast
Ġjed -> Ġwhatsoever
flughafen -> Air
vermittlung -> employment
teilung -> ignment
ĠInterpretation -> Ġinterpretation
Ġimport -> Ġimported
hansa -> irl
genehmigungen -> exempt
ĠAuflist -> Ġlists
Ġverschwunden -> Ġdisappeared
ĠFlyers -> ĠFlyers
Kanal -> Channel
Ġlehr -> Ġteachers
Ġnahelie -> Ġconvenient
gener -> Generally
mitarbeiter -> staff
This approach helped a lot in a trial run, though a full comparison was not possible due to budget and time constraints. You can pass the WTE matrix via the wte_path
to the training script. Credit to this blogpost for the idea of initializing GPT2 from English weights.
3. Tokenize the corpus using prepare/tokenize_text.py
. This generates files for train and validation tokens in JSON Lines format.
4. Run the training script train.py
! run.sh
shows how this was executed for the full run with config configs/tpu_large.json
.
📄 License
GerPT2 is licensed under the MIT License.
📚 Documentation
Citing
Please cite GerPT2 as follows:
@misc{Minixhofer_GerPT2_German_large_2020,
author = {Minixhofer, Benjamin},
doi = {10.5281/zenodo.5509984},
month = {12},
title = {{GerPT2: German large and small versions of GPT2}},
url = {https://github.com/bminixhofer/gerpt2},
year = {2020}
}
Acknowledgements
Thanks to Hugging Face for providing awesome tools and infrastructure.
Huge thanks to Artus Krohn - Grimberghe at LYTiQ for sponsoring the resources used for training, making this project possible.