TinyLlama Open-Source Small Model - Ideal for Resource-Constrained Environments, Free Deployment and Highly Practical

Tinyllama 1.1B Intermediate Step 1195k Token 2.5T

Developed by TinyLlama

TinyLlama is a compact 1.1B-parameter Llama model pretrained on 3 trillion tokens, designed for resource-constrained environments.

Large Language Model

Transformers

EnglishOpen Source License:Apache-2.0 #Lightweight Llama #Efficient Pretraining #Multi-task General Purpose

Downloads 419

Release Time : 12/11/2023

Model Overview

The TinyLlama project aims to pretrain a 1.1B-parameter Llama model, completed in 90 days using 16 A100-40G GPUs. The model adopts the same architecture and tokenizer as Llama 2, making it suitable for various Llama-based open-source projects.

Model Features

Efficient Training

Completed pretraining on 3 trillion tokens in 90 days using 16 A100-40G GPUs.

Compact Structure

With only 1.1B parameters, it is suitable for applications with limited computational and memory resources.

Compatibility

Adopts the same architecture and tokenizer as Llama 2, enabling plug-and-play use in Llama-based open-source projects.

Model Capabilities

Text Generation

Language Understanding

Use Cases

Natural Language Processing

Text Generation

Generate coherent English text

Language Understanding

Understand and answer English questions

Model	Pretrain Tokens	HellaSwag	Obqa	WinoGrande	ARC_c	ARC_e	boolq	piqa	avg
Pythia - 1.0B	300B	47.16	31.40	53.43	27.05	48.99	60.83	69.21	48.30
TinyLlama - 1.1B - intermediate - step - 50K - 104b	103B	43.50	29.80	53.28	24.32	44.91	59.66	67.30	46.11
TinyLlama - 1.1B - intermediate - step - 240k - 503b	503B	49.56	31.40	55.80	26.54	48.32	56.91	69.42	48.28
TinyLlama - 1.1B - intermediate - step - 480k - 1007B	1007B	52.54	33.40	55.96	27.82	52.36	59.54	69.91	50.22
TinyLlama - 1.1B - intermediate - step - 715k - 1.5T	1.5T	53.68	35.20	58.33	29.18	51.89	59.08	71.65	51.29
TinyLlama - 1.1B - intermediate - step - 955k - 2T	2T	54.63	33.40	56.83	28.07	54.67	63.21	70.67	51.64
TinyLlama - 1.1B - intermediate - step - 1195k - token - 2.5T	2.5T	58.96	34.40	58.72	31.91	56.78	63.21	73.07	53.86

Property	Details
Model Type	TinyLlama-1.1B
Training Data	cerebras/SlimPajama - 627B, bigcode/starcoderdata

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご

Tinyllama 1.1B Intermediate Step 1195k Token 2.5T

Model Overview

Model Features

Model Capabilities

Use Cases

🚀 TinyLlama-1.1B

✨ Features

This Collection

Eval

📄 License