Tinyllama 1.1B Chat V0.4 GGUF

🚀 TinyLlama/TinyLlama-1.1B-Chat-v0.4-GGUF

This project provides quantized GGUF model files for TinyLlama-1.1B-Chat-v0.4 from TinyLlama. These quantized models offer a more memory - efficient option for various applications.

✨ Features

• Quantized Models: Available in multiple quantization methods such as q2_k, q3_k_m, q4_k_m, q5_k_m, q6_k, and q8_0, providing different trade - offs between model size and performance.
• Compatibility: Adopts the same architecture and tokenizer as Llama 2, making it plug - and - play in many open - source projects based on Llama.
• Compactness: With only 1.1B parameters, it is suitable for applications with limited computation and memory resources.

📦 Installation

The installation mainly involves setting up the necessary Python libraries. You will need transformers>=4.31. You can install it using the following command:

pip install transformers>=4.31

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer
import transformers 
import torch
model = "PY007/TinyLlama-1.1B-Chat-v0.4"
tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

CHAT_EOS_TOKEN_ID = 32002

prompt = "How to get in a good university?"
formatted_prompt = (
    f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"
)


sequences = pipeline(
    formatted_prompt,
    do_sample=True,
    top_k=50,
    top_p = 0.9,
    num_return_sequences=1,
    repetition_penalty=1.1,
    max_new_tokens=1024,
    eos_token_id=CHAT_EOS_TOKEN_ID,
)

for seq in sequences:
    print(f"Result: {seq['generated_text']}")

📚 Documentation

Model Information

Available Quantized Models

Original Model Card

The TinyLlama project aims to pretrain a 1.1B Llama model on 3 trillion tokens. With some proper optimization, it can achieve this within a span of "just" 90 days using 16 A100 - 40G GPUs 🚀🚀. The training started on 2023 - 09 - 01.

This is the chat model finetuned on top of TinyLlama/TinyLlama-1.1B-intermediate-step-715k-1.5T. The dataset used is OpenAssistant/oasst_top1_2023-08-25 following the chatml format.

More Information

Do check the TinyLlama GitHub page for more information.

📄 License

This project is licensed under the apache - 2.0 license.