Distilgpt2 Tiny Conversational

🚀 distilgpt2-tiny-conversational

This is a fine - tuned version of distilgpt2 on a parsed version of Wizard of Wikipedia. It uses the Persona alpha/beta framework and is designed for use with ai-msgbot. This model achieves a loss of 2.2461 on the evaluation set.

✨ Features

• It is a basic dialogue model for conversation and can be used as a chatbot.
• Check out a simple demo here

📦 Installation

No installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

You can use the provided widget examples to interact with the model. Here are some examples:

• walk:

I know you're tired, but can we go for another walk this evening?
person beta:

• activities:

Have you done anything exciting lately?
person beta:

• grocery:

hey - do you have a favorite grocery store around here?
person beta:

• dinner:

Can you take me for dinner somewhere nice this time?
person beta:

• social media:

What's your favorite form of social media?
person beta:

• greeting:

Hi, how are you?
person beta:

• sister:

I am the best; my sister is the worst. What am I?
person beta:

• alligator:

What do you call an alligator who's just had surgery to remove his left arm?
person beta:

• dollar:

A man walks into a bar and asks for a drink. The bartender asks for $10, and he pays him $1. What did he pay him with?
person beta:

• mailbox:

What did I say was in the mailbox when it was actually in the cabinet?
person beta:

• language:

My friend says that she knows every language, but she doesn't speak any of them.. what's wrong with her?
person beta:

Inference Parameters

The following parameters are used for inference:

📚 Documentation

Intended Uses & Limitations

• It is designed for integrating with this repo: ai-msgbot
• The model generates whole conversations between two entities, person alpha and person beta. These entity names are used functionally as custom <bos> tokens to extract when one response ends and another begins.

Training and Evaluation Data

• The training data is wizard of Wikipedia parsed, from parlAI

Training Procedure

• It uses deepspeed + huggingface trainer. An example notebook is in ai-msgbot

Training Hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e - 05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• distributed_type: multi - GPU
• gradient_accumulation_steps: 4
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon = 1e - 08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 30

Training Results

Framework Versions

• Transformers 4.16.1
• Pytorch 1.10.0+cu111
• Tokenizers 0.11.0

📄 License

This project is licensed under the Apache - 2.0 license.