Tinystories 33M

🚀 TinyStories-33M Model

This model is trained on the TinyStories dataset and is based on the GPT - Neo architecture, offering text generation capabilities.

🚀 Quick Start

The model is trained on the TinyStories Dataset. For more details, refer to this paper. It's based on the GPT - Neo architecture.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig

model = AutoModelForCausalLM.from_pretrained('roneneldan/TinyStories-33M')
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-125M")
prompt = "Once upon a time there was"
input_ids = tokenizer.encode(prompt, return_tensors="pt")

# Generate completion
output = model.generate(input_ids, max_length = 1000, num_beams=1)

# Decode the completion
output_text = tokenizer.decode(output[0], skip_special_tokens=True)

# Print the generated text
print(output_text)

🔧 Technical Details

The following hyperparameters were used to train this model:

lr = 5e-4,
lr_schedule = constant, 
wd=0.1,
adam_beta1=0.9, adam_beta2 = 0.95,
context_length=512,
batch_size=80,
gradient_accumulation_steps=16

📄 License

This project is licensed under the MIT license.

📋 Model Information