Litellama 460M 1T

🚀 LiteLlama: Reduced-Scale Llama

We present an open-source reproduction of Meta AI's LLaMa 2 with significantly reduced model sizes.

This project offers an open - source replication of Meta AI's LLaMa 2. Notably, it features substantially reduced model sizes. For instance, LiteLlama - 460M - 1T has 460M parameters and is trained with 1T tokens.

✨ Features

• Reduced Scale: Significantly smaller model sizes compared to the original LLaMa 2.
• Open - Source: An open - source reproduction for community use.

📦 Installation

This section is not provided in the original README, so it is skipped.

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_path = 'ahxt/LiteLlama-460M-1T'

model = AutoModelForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
model.eval()

prompt = 'Q: What is the largest bird?\nA:'
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
tokens = model.generate(input_ids, max_length=20)
print( tokenizer.decode(tokens[0].tolist(), skip_special_tokens=True) )
# Q: What is the largest bird?\nA: The largest bird is a black-headed gull.

📚 Documentation

Dataset and Tokenization

We train our models on part of RedPajama dataset. We use the GPT2Tokenizer to tokenize the text.

Training Details

The model was trained with ~1T tokens (0.98T). num of tokens = stepslengthbatch_size = 4996791024192 = 98240888832≈0.98T.

The training curve is at this WandB project.

Evaluation

MMLU Task Evaluation

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

🔧 Technical Details

The model's training process involves specific calculations for the number of tokens used, and the training curve can be tracked through the provided WandB project link.

📄 License

This model is released under the MIT License.

📞 Contact

This model was developed by Xiaotian Han from Texas A&M University at the DATA Lab under the supervision of Prof. Xia "Ben" Hu.