TinyLlama-1.1B-32k Open Source Model - Free to Use, Superb at Handling Ultra-Long Context Content

Tinyllama 1.1B 32k

Developed by Doctor-Shotgun

A 32k-context fine-tuned version based on TinyLlama-1.1B, achieving long-context processing capability by increasing rope theta

Large Language Model

Transformers

EnglishOpen Source License:Apache-2.0 #Long-context reasoning #Speculative decoding optimization #Efficient fine-tuning

Downloads 51

Release Time : 12/29/2023

Model Overview

This is a language model optimized for long contexts, supporting 32k context length by adjusting rope frequency base, suitable for use as a speculative decoding model

Model Features

Long-context support

Supports 32k context length by adjusting rope theta parameter

Efficient inference

Quantized version can run on a single A6000 GPU, suitable for speculative decoding

Optimized pretraining

Pretrained on RedPajama-Data-1T-Sample dataset with 32k context length

Model Capabilities

Long text generation

Code generation

Text understanding

Use Cases

Code generation

Programming assistance

Used for generating and completing code

HumanEval evaluation shows Pass@1 reaching 0.0829

Long text processing

Long document analysis

Processes text content up to 32k tokens

Perplexity of 7.1338 at 32768 length

Model	2048	4096	8192	16384	32768
TinyLlama-1.1B	8.5633	208.3586	863.7507	1600.5021	6981.9021
TinyLlama-1.1B-32k	8.6548	7.8339	7.4904	7.3674	7.1338

Model	Pass@1	Pass@10
TinyLlama-1.1B	0.0841	0.1524
TinyLlama-1.1B (NTK alpha=7.7)	0.0598	0.1098
TinyLlama-1.1B-32k-ckpt-554	0.0732	0.1402
TinyLlama-1.1B-32k	0.0829	0.1524

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご

Tinyllama 1.1B 32k

Model Overview

Model Features

Model Capabilities

Use Cases

🚀 TinyLlama-1.1B-32k

📚 Documentation

Wikitext (wikitext-2-raw-v1_train) Perplexity (64 rows) as evaluated via exllamav2:

Evaluation on HumanEval by turboderp:

📄 License