dansk-gpt-wiki: An Open-Source Danish Model - Leveraging Wiki Data to Complete Practical Tasks such as Text Generation

Dansk Gpt Wiki

Developed by flax-community

A Danish GPT2-style model trained using the Flax CLM pipeline, based on the Danish portion of the wiki40b dataset.

Large Language Model Other#Danish language generation #Wikipedia training #Nordic language model

Downloads 14.89k

Release Time : 3/2/2022

Model Overview

This is a Danish language model based on the GPT2 architecture, specifically trained on Danish Wikipedia content, suitable for Danish text generation tasks.

Model Features

Danish language optimization

Specifically trained on Danish Wikipedia content, offering better understanding and generation capabilities for Danish text.

Trained with Flax/Jax

Efficiently trained on TPUs using the Flax and Jax frameworks.

Wikipedia data training

Trained using the Danish portion of the wiki40b dataset, containing rich encyclopedic knowledge.

Model Capabilities

Danish text generation

Language modeling

Text continuation

Use Cases

Content generation

Wikipedia-style text generation

Generate Danish text resembling Wikipedia entries

Danish writing assistance

Assist users in completing Danish writing tasks

Education

Danish learning tool

Serve as a language model aid for Danish learners

🚀 GPT2-svenska-wikipedia

A Danish GPT2 style model trained using Flax CLM pipeline on the Danish part of the wiki40b dataset.

🚀 Quick Start

This is a Danish GPT2 style model trained using the Flax CLM pipeline on the Danish part of the wiki40b dataset. You can access the dataset via wiki40b.

✨ Features

This model is part of a series of models trained on TPU with Flax Jax during the Huggingface Flax/Jax challenge.

💻 Usage Examples

Data cleaning and preprocessing

The data was cleaned and preprocessed using the following script. Make sure to install depencies for beam_runner to make the dataset work.

from datasets import load_dataset
def load_and_clean_wiki():
    dataset = load_dataset('wiki40b', 'da', beam_runner='DirectRunner', split="train")
    #dataset = load_dataset('wiki40b', 'sv', beam_runner='DirectRunner')
    dataset = dataset.remove_columns(['wikidata_id', 'version_id'])
    filtered_dataset = dataset.map(filter_wikipedia)
    # filtered_dataset[:3]
    # print(filtered_dataset[:3])
    return filtered_dataset

def filter_wikipedia(batch):
    batch["text"] = " ".join(batch["text"].split("\
_START_SECTION_\
"))
    batch["text"] = " ".join(batch["text"].split("\
_START_ARTICLE_\
"))
    batch["text"] = " ".join(batch["text"].split("\
_START_ARTICLE_\
"))
    batch["text"] = " ".join(batch["text"].split("\
_START_PARAGRAPH_\
"))
    batch["text"] = " ".join(batch["text"].split("_NEWLINE_"))
    batch["text"] = " ".join(batch["text"].split("\xa0"))
    return batch

Training script

The following training script was used to train the model.

./run_clm_flax.py     --output_dir="${MODEL_DIR}"     --model_type="gpt2"     --config_name="${MODEL_DIR}"     --tokenizer_name="${MODEL_DIR}"     --dataset_name="wiki40b"     --dataset_config_name="da"     --do_train --do_eval     --block_size="512"     --per_device_train_batch_size="64"     --per_device_eval_batch_size="64"     --learning_rate="5e-3" --warmup_steps="1000"     --adam_beta1="0.9" --adam_beta2="0.98" --weight_decay="0.01"     --overwrite_output_dir     --num_train_epochs="20"     --logging_steps="500"     --save_steps="1000"     --eval_steps="2500"     --push_to_hub

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご