GPT2-medium-dutch-embeddings Open-source Dutch Model - Meet the Requirements of Dutch Text Processing

Gpt2 Medium Dutch Embeddings

Developed by GroNLP

A Dutch language model based on the medium-scale GPT-2 version, with only the vocabulary embedding layer retrained for Dutch adaptation.

Large Language Model Other#Dutch language generation #Vocabulary embedding adaptation #Medium-scale GPT-2

Downloads 27

Release Time : 3/2/2022

Model Overview

This model adapts the English GPT-2 model to Dutch by retraining the vocabulary embedding layer while preserving the weights of the original Transformer layers.

Model Features

Vocabulary Embedding Adaptation

Only the vocabulary embedding layer is retrained to adapt to Dutch vocabulary while preserving the weights of the original Transformer layers.

Model Recycling

Effectively utilizes parameters from existing pre-trained models by retraining the vocabulary layer instead of training from scratch.

Medium-scale

Based on the medium-scale version of GPT-2 (gpt2-medium), providing stronger language understanding capabilities.

Model Capabilities

Dutch text generation

Foundation for language model fine-tuning

Use Cases

Natural Language Processing

Dutch Text Generation

Generate coherent Dutch text

Downstream Task Fine-tuning

Serves as a base model for fine-tuning Dutch NLP tasks

🚀 GPT-2 recycled for Dutch (medium, adapted lexical embeddings)

This model is based on the medium OpenAI GPT - 2, with its Transformer layer weights identical to the original English model, but the lexical layer retrained for a Dutch vocabulary.

🚀 Quick Start

💻 Usage Examples

Basic Usage

from transformers import pipeline

pipe = pipeline("text-generation", model="GroNLP/gpt2-medium-dutch-embeddings")

Advanced Usage

from transformers import AutoTokenizer, AutoModel, TFAutoModel

tokenizer = AutoTokenizer.from_pretrained("GroNLP/gpt2-medium-dutch-embeddings")
model = AutoModel.from_pretrained("GroNLP/gpt2-medium-dutch-embeddings")  # PyTorch
model = TFAutoModel.from_pretrained("GroNLP/gpt2-medium-dutch-embeddings")  # Tensorflow

✨ Features

This model is based on the medium OpenAI GPT - 2 (gpt2-medium) model. The Transformer layer weights in this model are identical to the original English model, but the lexical layer has been retrained for a Dutch vocabulary. For details, check out our paper on arXiv and the code on Github.

📚 Documentation

Related models

Dutch

gpt2-small-dutch-embeddings: Small model size with only retrained lexical embeddings.
gpt2-small-dutch: Small model size with retrained lexical embeddings and additional fine - tuning of the full model. (Recommended)
gpt2-medium-dutch-embeddings: Medium model size with only retrained lexical embeddings.

Italian

gpt2-small-italian-embeddings: Small model size with only retrained lexical embeddings.
gpt2-small-italian: Small model size with retrained lexical embeddings and additional fine - tuning of the full model. (Recommended)
gpt2-medium-italian-embeddings: Medium model size with only retrained lexical embeddings.

📄 License

BibTeX entry

@misc{devries2020good,
      title={As good as new. How to successfully recycle English GPT-2 to make models for other languages}, 
      author={Wietse de Vries and Malvina Nissim},
      year={2020},
      eprint={2012.05628},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご