language:

• en license: apache-2.0 tags:
• text-generation base_model: JackFram/llama-160m datasets:
• ehartford/wizard_vicuna_70k_unfiltered
• totally-not-an-llm/EverythingLM-data-V3
• Open-Orca/SlimOrca-Dedup
• databricks/databricks-dolly-15k
• THUDM/webglm-qa widget:
• messages:
  • role: system content: You are a helpful assistant, who answers with empathy.
  • role: user content: Got a question for you!
  • role: assistant content: Sure! What's it?
  • role: user content: Why do you love cats so much!? 🐈
• messages:
  • role: system content: You are a helpful assistant who answers user's questions with empathy.
  • role: user content: Who is Mona Lisa?
• messages:
  • role: system content: You are a helpful assistant who provides concise responses.
  • role: user content: Heya!
  • role: assistant content: Hi! How may I help you today?
  • role: user content: I need to build a simple website. Where should I start learning about web development?
• messages:
  • role: user content: Invited some friends to come home today. Give me some ideas for games to play with them!
• messages:
  • role: system content: You are a helpful assistant who answers user's questions with details and curiosity.
  • role: user content: What are some potential applications for quantum computing?
• messages:
  • role: system content: You are a helpful assistant who gives creative responses.
  • role: user content: Write the specs of a game about mages in a fantasy world.
• messages:
  • role: system content: You are a helpful assistant who answers user's questions with details.
  • role: user content: Tell me about the pros and cons of social media.
• messages:
  • role: system content: You are a helpful assistant who answers user's questions with confidence.
  • role: user content: What is a dog?
  • role: assistant content: A dog is a four-legged, domesticated animal that is a member of the class Mammalia, which includes all mammals. Dogs are known for their loyalty, playfulness, and ability to be trained for various tasks. They are also used for hunting, herding, and as service animals.
  • role: user content: What is the color of an apple? inference: parameters: max_new_tokens: 250 penalty_alpha: 0.5 top_k: 4 repetition_penalty: 1.01 model-index:
• name: Llama-160M-Chat-v1 results:
  • task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics:
    • type: acc_norm value: 24.74 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics:
    • type: acc_norm value: 35.29 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics:
    • type: acc value: 26.13 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics:
    • type: mc2 value: 44.16 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics:
    • type: acc value: 51.3 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics:
    • type: acc value: 0.0 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: IFEval (0-Shot) type: HuggingFaceH4/ifeval args: num_few_shot: 0 metrics:
    • type: inst_level_strict_acc and prompt_level_strict_acc value: 15.75 name: strict accuracy source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: BBH (3-Shot) type: BBH args: num_few_shot: 3 metrics:
    • type: acc_norm value: 3.17 name: normalized accuracy source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: MATH Lvl 5 (4-Shot) type: hendrycks/competition_math args: num_few_shot: 4 metrics:
    • type: exact_match value: 0.0 name: exact match source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: GPQA (0-shot) type: Idavidrein/gpqa args: num_few_shot: 0 metrics:
    • type: acc_norm value: 1.01 name: acc_norm source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: MuSR (0-shot) type: TAUR-Lab/MuSR args: num_few_shot: 0 metrics:
    • type: acc_norm value: 3.17 name: acc_norm source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: MMLU-PRO (5-shot) type: TIGER-Lab/MMLU-Pro config: main split: test args: num_few_shot: 5 metrics:
    • type: acc value: 1.51 name: accuracy source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 name: Open LLM Leaderboard

A Llama Chat Model of 160M Parameters

• Base model: JackFram/llama-160m
• Datasets:
  • ehartford/wizard_vicuna_70k_unfiltered
  • totally-not-an-llm/EverythingLM-data-V3
  • Open-Orca/SlimOrca-Dedup
  • databricks/databricks-dolly-15k
  • THUDM/webglm-qa
• Availability in other ML formats:
  • GGUF: Felladrin/gguf-Llama-160M-Chat-v1
  • ONNX: Felladrin/onnx-Llama-160M-Chat-v1
  • MLC: Felladrin/mlc-q4f16-Llama-160M-Chat-v1
  • MLX: mlx-community/Llama-160M-Chat-v1-4bit-mlx

Recommended Prompt Format

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant

Recommended Inference Parameters

penalty_alpha: 0.5
top_k: 4
repetition_penalty: 1.01

Usage Example

from transformers import pipeline

generate = pipeline("text-generation", "Felladrin/Llama-160M-Chat-v1")

messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant who answers user's questions with details and curiosity.",
    },
    {
        "role": "user",
        "content": "What are some potential applications for quantum computing?",
    },
]

prompt = generate.tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

output = generate(
    prompt,
    max_new_tokens=1024,
    penalty_alpha=0.5,
    top_k=4,
    repetition_penalty=1.01,
)

print(output[0]["generated_text"])

Old Open LLM Leaderboard Evaluation Results

New Open LLM Leaderboard Evaluation Results

Detailed results can be found here