Tinychat 1776K

🚀 raincandy-u/TinyChat-1776K

A tiny Language Model (LM) trained from scratch on the TinyChat dataset, aiming to achieve natural responses with the smallest possible model.

This model is trained using a dataset of 3 - year - old children level English conversations. It's important to note that it has no world knowledge, so you shouldn't ask it any intellectual questions.

🚀 Quick Start

This section provides an overview of the model and its limitations. Remember, since it lacks world knowledge, avoid asking intellectual questions.

✨ Features

• Trained from scratch on the TinyChat dataset.
• Aims to achieve natural responses with a minimal model.
• Uses a dataset of 3 - year - old children level English conversations.

📚 Documentation

🔧 Model Spec

The following code shows the configuration of the model:

config = AutoConfig.for_model(
    model_type="llama",
    hidden_size=192,
    intermediate_size=640,
    num_attention_heads=16,
    num_hidden_layers=3,
    num_key_value_heads=4,
    tie_word_embeddings=True,
    vocab_size=2048,
    max_position_embeddings=256
)

💻 Usage Examples

🔍 Basic Usage

Here is a template for using the model:

# Template example
print("<A>Hi, Tom. How are you? <end>")
print("<B>I'm fine, thank you. And you? <end>")
print("<A>Fine. What's your favorite color? <end>")
print("<B>My favorite color is black. <end>")
print("<A>Do you like cats? <end>")
print("<B>")

🚀 Advanced Usage

The example output for the above template is:

print("Yes, I do. I like it too. They are good for me.")

⚙️ Generation Param

When generating text, you can use the following parameters:

top_k=40,
top_p=0.8,
temperature=1

📄 License

This project is licensed under the Apache - 2.0 license.

📋 Information Table

⚠️ Important Note

⚠️ Important Note

The model has no world knowledge, so you should not ask it any intellectual questions.

💡 Usage Tip

💡 Usage Tip

Use the provided template and generation parameters to get the best results from the model.