Llama 3 3 Nemotron Super 49B V1 GGUF

🚀 Llama-3_3-Nemotron-Super-49B-v1 GGUF Models

This project offers a series of GGUF models based on Llama-3, featuring ultra-low-bit quantization techniques. These models are designed to balance accuracy and memory efficiency, catering to various hardware and deployment scenarios.

🚀 Quick Start

If you find these models useful, please click like ❤. Also, you can test the Network Monitor Assistant at 👉 Network Monitor Assitant. Click the chat icon (bottom right of the main and dashboard pages), choose a LLM, and toggle between the LLM Types TurboLLM -> FreeLLM -> TestLLM.

✨ Features

Ultra-Low-Bit Quantization with IQ-DynamicGate (1 - 2 bit)

• Precision - Adaptive Quantization: Our latest method introduces precision - adaptive quantization for ultra - low - bit models (1 - 2 bit), proven effective on Llama - 3 - 8B. It uses layer - specific strategies to preserve accuracy while maintaining extreme memory efficiency.
• Benchmark - Proven Improvements: All tests were conducted on Llama - 3 - 8B - Instruct using a standard perplexity evaluation pipeline, a 2048 - token context window, and the same prompt set across all quantizations.

Key Improvements in Quantization

• 🔥 IQ1_M shows a massive 43.9% perplexity reduction (from 27.46 to 15.41).
• 🚀 IQ2_S cuts perplexity by 36.9% while adding only 0.2GB.
• ⚡ IQ1_S maintains 39.7% better accuracy despite 1 - bit quantization.

Trade - offs

• All variants have modest size increases (0.1 - 0.3GB).
• Inference speeds remain comparable (<5% difference).

When to Use These Models

• 📌 Fitting models into GPU VRAM
• ✔ Memory - constrained deployments
• ✔ Cpu and Edge Devices where 1 - 2bit errors can be tolerated
• ✔ Research into ultra - low - bit quantization

Choosing the Right Model Format

• BF16 (Brain Float 16): Use if BF16 acceleration is available. It's a 16 - bit floating - point format for faster computation with good precision, suitable for high - performance inference with reduced memory footprint.
• F16 (Float 16): More widely supported than BF16. It has high precision but a smaller range of values than BF16, working on most devices with FP16 acceleration support.
• Quantized Models (Q4_K, Q6_K, Q8, etc.): For CPU & Low - VRAM Inference. Lower - bit models (Q4_K) are best for minimal memory usage, while higher - bit models (Q6_K, Q8_0) offer better accuracy.
• Very Low - Bit Quantization (IQ3_XS, IQ3_S, IQ3_M, Q4_K, Q4_0): Optimized for extreme memory efficiency, ideal for low - power devices or large - scale deployments with memory constraints.

📚 Documentation

Quantization Performance Comparison (Llama - 3 - 8B)

Key:

• PPL = Perplexity (lower is better)
• Δ PPL = Percentage change from standard to DynamicGate
• Speed = Inference time (CPU avx2, 2048 token context)
• Size differences reflect mixed quantization overhead

Summary Table: Model Format Selection

📦 Included Files & Details

Llama-3_3-Nemotron-Super-49B-v1-bf16.gguf

• Model weights are preserved in BF16. Use this if you want to requantize the model into a different format, and it's best if your device supports BF16 acceleration.

Llama-3_3-Nemotron-Super-49B-v1-f16.gguf

• Model weights are stored in F16. Use if your device supports FP16, especially if BF16 is not available.

Llama-3_3-Nemotron-Super-49B-v1-bf16-q8_0.gguf

• Output & embeddings remain in BF16, and all other layers are quantized to Q8_0. Use if your device supports BF16 and you want a quantized version.

Llama-3_3-Nemotron-Super-49B-v1-f16-q8_0.gguf

• Output & embeddings remain in F16, and all other layers are quantized to Q8_0.

Llama-3_3-Nemotron-Super-49B-v1-q4_k.gguf

• Output & embeddings are quantized to Q8_0, and all other layers are quantized to Q4_K. Good for CPU inference with limited memory.

Llama-3_3-Nemotron-Super-49B-v1-q4_k_s.gguf

• The smallest Q4_K variant, using less memory at the cost of accuracy. Best for very low - memory setups.

Llama-3_3-Nemotron-Super-49B-v1-q6_k.gguf

• Output & embeddings are quantized to Q8_0, and all other layers are quantized to Q6_K.

Llama-3_3-Nemotron-Super-49B-v1-q8_0.gguf

• A fully Q8 quantized model for better accuracy. Requires more memory but offers higher precision.

Llama-3_3-Nemotron-Super-49B-v1-iq3_xs.gguf

• IQ3_XS quantization, optimized for extreme memory efficiency. Best for ultra - low - memory devices.

Llama-3_3-Nemotron-Super-49B-v1-iq3_m.gguf

• IQ3_M quantization, offering a medium block size for better accuracy. Suitable for low - memory devices.

Llama-3_3-Nemotron-Super-49B-v1-q4_0.gguf

• Pure Q4_0 quantization, optimized for ARM devices. Best for low - memory environments. Prefer IQ4_NL for better accuracy.

📄 License

This project is under the nvidia - open - model - license.

🔧 Technical Details

Method of Quantization

• Dynamic Precision Allocation:
  • First/Last 25% of layers → IQ4_XS (selected layers)
  • Middle 50% → IQ2_XXS/IQ3_S (increase efficiency)
• Critical Component Protection:
  • Embeddings/output layers use Q5_K, which reduces error propagation by 38% vs standard 1 - 2bit.

What I'm Testing

I'm experimenting with function calling against my network monitoring service using small open - source models. I'm exploring the question "How small can it go and still function".

The other Available AI Assistants

• 🟢 TurboLLM – Uses gpt - 4o - mini, which is fast! Note that tokens are limited since OpenAI models are pricey, but you can Login or Download the Free Network Monitor agent to get more tokens. Alternatively, use the TestLLM.
• 🔵 HugLLM – Runs open - source Hugging Face models fast. It runs small models (≈8B) hence lower quality. Get 2x more tokens (subject to Hugging Face API availability).
• 🟡 TestLLM – Runs the current testing model using llama.cpp on 6 threads of a Cpu VM (Should take about 15s to load. Inference speed is quite slow and it only processes one user prompt at a time—still working on scaling!). If you're curious, I'd be happy to share how it works!