Olympiccoder 7B GGUF

🚀 OlympicCoder-7B GGUF Models

OlympicCoder-7B GGUF models offer ultra-low-bit quantization with IQ-DynamicGate (1 - 2 bit), providing high memory efficiency while maintaining accuracy. These models are suitable for various hardware and memory constraints, and are useful for competitive coding benchmarks.

✨ Features

Ultra-Low-Bit Quantization with IQ-DynamicGate (1 - 2 bit)

• Precision-adaptive quantization: Our latest quantization method introduces precision-adaptive quantization for ultra-low-bit models (1 - 2 bit), with benchmark-proven improvements on Llama-3-8B.
• Layer-specific strategies: This approach uses layer-specific strategies to preserve accuracy while maintaining extreme memory efficiency.

Benchmark Context

• Testing environment: All tests were conducted on Llama-3-8B-Instruct using a standard perplexity evaluation pipeline, a 2048-token context window, and the same prompt set across all quantizations.

Method

• Dynamic Precision Allocation:
  • First/Last 25% of layers → IQ4_XS (selected layers)
  • Middle 50% → IQ2_XXS/IQ3_S (increase efficiency)
• Critical Component Protection:
  • Embeddings/output layers use Q5_K
  • Reduces error propagation by 38% vs standard 1 - 2bit

Quantization Performance Comparison (Llama-3-8B)

Key Improvements:

• 🔥 IQ1_M shows a massive 43.9% perplexity reduction (27.46 → 15.41)
• 🚀 IQ2_S cuts perplexity by 36.9% while adding only 0.2GB
• ⚡ IQ1_S maintains 39.7% better accuracy despite 1-bit quantization

Tradeoffs:

• All variants have modest size increases (0.1 - 0.3GB)
• Inference speeds remain comparable (<5% difference)

When to Use These Models

• 📌 Fitting models into GPU VRAM
• ✔ Memory-constrained deployments
• ✔ Cpu and Edge Devices where 1 - 2bit errors can be tolerated
• ✔ Research into ultra-low-bit quantization

📦 Installation

No installation steps provided in the original document.

💻 Usage Examples

Basic Usage

# pip install transformers
# pip install accelerate

import torch
from transformers import pipeline

pipe = pipeline("text-generation", model="open-r1/OlympicCoder-7B", torch_dtype=torch.bfloat16, device_map="auto")

# We use the tokenizer's chat template to format each message - see https://huggingface.co/docs/transformers/main/en/chat_templating
messages = [
    {"role": "user", "content": "Write a python program to calculate the 10th Fibonacci number"},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt, max_new_tokens=8000, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
#<|im_start|>user
#Write a python program to calculate the 10th fibonacci number<|im_end|>
#<|im_start|>assistant
#<think>Okay, I need to write a Python program that calculates the 10th Fibonacci number. Hmm, the Fibonacci sequence starts with 0 and 1. Each subsequent number is the sum of the two preceding ones. So the sequence goes: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, and so on. ...

📚 Documentation

Choosing the Right Model Format

Selecting the correct model format depends on your hardware capabilities and memory constraints.

Summary Table: Model Format Selection

Included Files & Details

• OlympicCoder-7B-bf16.gguf: Model weights preserved in BF16. Use this if you want to requantize the model into a different format. Best if your device supports BF16 acceleration.
• OlympicCoder-7B-f16.gguf: Model weights stored in F16. Use if your device supports FP16, especially if BF16 is not available.
• OlympicCoder-7B-bf16-q8_0.gguf: Output & embeddings remain in BF16. All other layers quantized to Q8_0. Use if your device supports BF16 and you want a quantized version.
• OlympicCoder-7B-f16-q8_0.gguf: Output & embeddings remain in F16. All other layers quantized to Q8_0.
• OlympicCoder-7B-q4_k.gguf: Output & embeddings quantized to Q8_0. All other layers quantized to Q4_K. Good for CPU inference with limited memory.
• OlympicCoder-7B-q4_k_s.gguf: Smallest Q4_K variant, using less memory at the cost of accuracy. Best for very low-memory setups.
• OlympicCoder-7B-q6_k.gguf: Output & embeddings quantized to Q8_0. All other layers quantized to Q6_K.
• OlympicCoder-7B-q8_0.gguf: Fully Q8 quantized model for better accuracy. Requires more memory but offers higher precision.
• OlympicCoder-7B-iq3_xs.gguf: IQ3_XS quantization, optimized for extreme memory efficiency. Best for ultra-low-memory devices.
• OlympicCoder-7B-iq3_m.gguf: IQ3_M quantization, offering a medium block size for better accuracy. Suitable for low-memory devices.
• OlympicCoder-7B-q4_0.gguf: Pure Q4_0 quantization, optimized for ARM devices. Best for low-memory environments. Prefer IQ4_NL for better accuracy.

Testing the Models

• How to test: Click the chat icon (bottom right on any page), choose an AI assistant type (TurboLLM, FreeLLM, TestLLM), and start testing.
• What I’m Testing: Pushing the limits of small open-source models for AI network monitoring, including function calling against live network services, and how small a model can go while still handling automated Nmap scans, quantum-readiness checks, and Metasploit integration.

Other Assistants

• TurboLLM: Uses gpt-4-mini for real-time network diagnostics and automated penetration testing (Nmap/Metasploit). Get more tokens by downloading our Free Network Monitor Agent.
• HugLLM: Open-source models (≈8B params), offers 2x more tokens than TurboLLM, and provides AI-powered log analysis. Runs on Hugging Face Inference API.

Example AI Commands to Test

• "Give me info on my websites SSL certificate"
• "Check if my server is using quantum safe encyption for communication"
• "Run a quick Nmap vulnerability test"

🔧 Technical Details

Model Card for OlympicCoder-7B

• Repository: https://github.com/huggingface/open-r1
• Blog post: https://huggingface.co/blog/open-r1/update-3

Model description

• Model type: A 7B parameter model fine-tuned on a decontaminated version of the codeforces dataset.
• Language(s) (NLP): Primarily English
• License: apache-2.0
• Finetuned from model: Qwen/Qwen2.5-Coder-7B-Instruct

Evaluation

• Benchmarks: Compared the performance of OlympicCoder models on two main benchmarks for competitive coding: LiveCodeBench and the 2024 International Olympiad in Informatics.
• Note: The OlympicCoder models were post-trained exclusively on C++ solutions generated by DeepSeek-R1. As a result, the performance on LiveCodeBench should be considered to be partially out-of-domain, since this expects models to output solutions in Python.

Training procedure

Training hyper-parameters

• dataset: open-r1/codeforces-cots
• learning_rate: 4.0e-5
• train_batch_size: 2
• seed: 42
• packing: false
• distributed_type: deepspeed-zero-3
• num_devices: 8
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_min_lr
• min_lr_rate: 0.1
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 10.0

📄 License

The model is licensed under the apache-2.0 license.

⚠️ Important Note

To ensure that the model consistently outputs a long chain-of-thought, we have edited the chat template to prefill the first assistant turn with a <think> token. As a result, the outputs from this model will not show the opening <think> token if you use the model's generate() method. To apply reinforcement learning with a format reward, either prepend the <think> token to the model's completions or amend the chat template to remove the prefill.

💡 Usage Tip

If you find these models useful, please click "Like"! Help test the AI-Powered Network Monitor Assistant with quantum-ready security checks at Free Network Monitor.