Gpt2 Spanish

🚀 GPT2-Spanish

GPT2-Spanish is a language generation model trained from scratch using 11.5GB of Spanish texts. It employs a Byte Pair Encoding (BPE) tokenizer specifically trained for this task, with parameters identical to the small version of the original OpenAI GPT2 model.

✨ Features

• Trained from scratch with 11.5GB of Spanish texts.
• Utilizes a Byte Pair Encoding (BPE) tokenizer trained for Spanish.
• Parameters match the small version of the original OpenAI GPT2 model.

📦 Installation

No installation steps were provided in the original document, so this section is skipped.

📚 Documentation

Corpus

This model was trained with a 11.5GB text corpus, which includes 3.5GB of Wikipedia articles and 8GB of books (covering narrative, short stories, theater, poetry, essays, and popularization).

Tokenizer

The texts are tokenized using a byte - level version of Byte Pair Encoding (BPE) (for Unicode characters) with a vocabulary size of 50257. The inputs are sequences of 1024 consecutive tokens.

This tokenizer was trained from scratch using the Spanish corpus because the tokenizers of English models were found to have limitations in capturing the semantic relations of Spanish due to the morphosyntactic differences between the two languages.

In addition to the special token "<|endoftext|>" used for text ending in OpenAI GPT - 2 models, the tokens "<|talk|>", "<|ax1|>", "<|ax2|>" (...) "<|ax9|>" were included to serve as prompts in future training.

Training

The model and tokenizer were trained using Hugging Face libraries on Google Colab servers with an Nvidia Tesla V100 GPU having 16GB of memory.

Authors

The model was trained by Alejandro Oñate Latorre (Spain) and Jorge Ortiz Fuentes (Chile), members of -Deep ESP-, an open - source community on Natural Language Processing in Spanish (https://t.me/joinchat/VoEp1bPrDYEexc6h).

Thanks to the community members who contributed funding for the initial tests.

Cautions

The model generates texts based on the patterns learned from the training corpus. Since these data were not filtered, the model may generate offensive or discriminatory content.

📄 License

This project is licensed under the MIT license.