Refact 1 6B Fim GGUF

🚀 Refact-1.6B

This is the Refact-1.6B model, which offers high - performance code generation and chat capabilities. After fine - tuning on generated data, it outperforms many other models in code - related tasks, and also shows good performance in chat scenarios.

🚀 Quick Start

Code Generation

# pip install -q transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "smallcloudai/Refact-1_6B-fim"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, trust_remote_code=True).to(device)

prompt = '<fim_prefix>def print_hello_world():\n    """<fim_suffix>\n    print("Hello world!")<fim_middle>'

inputs = tokenizer.encode(prompt, return_tensors="pt").to(device)
outputs = model.generate(inputs, max_length=100, temperature=0.2)
print("-"*80)
print(tokenizer.decode(outputs[0]))

Chat Usage

prompt_template = "<empty_output>SYSTEM {system}\n" \
                  "<empty_output>USER {query}\n" \
                  "<empty_output>ASSISTANT"
prompt = prompt_template.format(system="You are a programming assistant",
                                query="How do I sort a list in Python?")

✨ Features

• High - performance Code Generation: After fine - tuning, it beats many other models such as Replit 3b, Stability Code 3b in code generation tasks.
• Chat Capability: Can be used as a chat model, and shows good performance in comparison with other chat - specialized models.
• Multi - language Support: Although trained mainly on English text, it has exposure to multiple languages in code comments.
• Fill - in - the - Middle (FIM): Supports the FIM feature, which is useful for code completion in specific scenarios.

📦 Installation

There is no specific installation command provided in the original document. If you want to use the model, you can install the necessary libraries as shown in the quick start code, for example:

pip install -q transformers

💻 Usage Examples

Basic Usage

# pip install -q transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "smallcloudai/Refact-1_6B-fim"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, trust_remote_code=True).to(device)

prompt = '<fim_prefix>def print_hello_world():\n    """<fim_suffix>\n    print("Hello world!")<fim_middle>'

inputs = tokenizer.encode(prompt, return_tensors="pt").to(device)
outputs = model.generate(inputs, max_length=100, temperature=0.2)
print("-"*80)
print(tokenizer.decode(outputs[0]))

Advanced Usage

prompt_template = "<empty_output>SYSTEM {system}\n" \
                  "<empty_output>USER {query}\n" \
                  "<empty_output>ASSISTANT"
prompt = prompt_template.format(system="You are a programming assistant",
                                query="How do I sort a list in Python?")

📚 Documentation

Model Architecture

As described in more detail in the blog post, we used:

• ALiBi based attention
• LayerNorm instead of RMSNorm
• Multi Query Attention We also used LiON, flash attention, early dropout.

Pretraining

For the base model, we used our own dataset that contains code with permissive licenses only, and open text datasets. Filtering is the key to success of this model:

• We only used text in English.
• Only topics related to computer science.
• Applied heavy deduplication. The text to code proportion was 50:50, model trained for 1.2T tokens.

Finetuning

We tested our hypothesis that chat data should boost base model performance in FIM and regular left - to - right code completion. We found that just 15% of open code instruction - following datasets, that we filtered for quality, improves almost all metrics. Additionally, to improve FIM, we observed common failure modes, and prepared a synthetic dataset based on The Stack dedup v1.1 to address them.

🔧 Technical Details

Ultra - Low - Bit Quantization with IQ - DynamicGate (1 - 2 bit)

Our latest quantization method introduces precision - adaptive quantization for ultra - low - bit models (1 - 2 bit), with benchmark - proven improvements on Llama - 3 - 8B. This approach uses layer - specific strategies to preserve accuracy while maintaining extreme memory efficiency.

Benchmark Context

All tests conducted on Llama - 3 - 8B - Instruct using:

• Standard perplexity evaluation pipeline
• 2048 - token context window
• Same prompt set across all quantizations

Method

• Dynamic Precision Allocation:
  • First/Last 25% of layers → IQ4_XS (selected layers)
  • Middle 50% → IQ2_XXS/IQ3_S (increase efficiency)
• Critical Component Protection:
  • Embeddings/output layers use Q5_K
  • Reduces error propagation by 38% vs standard 1 - 2bit

Quantization Performance Comparison (Llama - 3 - 8B)

• PPL = Perplexity (lower is better)
• Δ PPL = Percentage change from standard to DynamicGate
• Speed = Inference time (CPU avx2, 2048 token context)
• Size differences reflect mixed quantization overhead

Choosing the Right Model Format

Selecting the correct model format depends on your hardware capabilities and memory constraints.

BF16 (Brain Float 16) – Use if BF16 acceleration is available

• A 16 - bit floating - point format designed for faster computation while retaining good precision.
• Provides similar dynamic range as FP32 but with lower memory usage.
• Recommended if your hardware supports BF16 acceleration (check your device's specs).
• Ideal for high - performance inference with reduced memory footprint compared to FP32.

F16 (Float 16) – More widely supported than BF16

• A 16 - bit floating - point high precision but with less of range of values than BF16.
• Works on most devices with FP16 acceleration support (including many GPUs and some CPUs).
• Slightly lower numerical precision than BF16 but generally sufficient for inference.

Quantized Models (Q4_K, Q6_K, Q8, etc.) – For CPU & Low - VRAM Inference

Quantization reduces model size and memory usage while maintaining as much accuracy as possible.

• Lower - bit models (Q4_K) → Best for minimal memory usage, may have lower precision.
• Higher - bit models (Q6_K, Q8_0) → Better accuracy, requires more memory.

Very Low - Bit Quantization (IQ3_XS, IQ3_S, IQ3_M, Q4_K, Q4_0)

These models are optimized for extreme memory efficiency, making them ideal for low - power devices or large - scale deployments where memory is a critical constraint.

• IQ3_XS: Ultra - low - bit quantization (3 - bit) with extreme memory efficiency.
  • Use case: Best for ultra - low - memory devices where even Q4_K is too large.
  • Trade - off: Lower accuracy compared to higher - bit quantizations.
• IQ3_S: Small block size for maximum memory efficiency.
  • Use case: Best for low - memory devices where IQ3_XS is too aggressive.
• IQ3_M: Medium block size for better accuracy than IQ3_S.
  • Use case: Suitable for low - memory devices where IQ3_S is too limiting.
• Q4_K: 4 - bit quantization with block - wise optimization for better accuracy.
  • Use case: Best for low - memory devices where Q6_K is too large.
• Q4_0: Pure 4 - bit quantization, optimized for ARM devices.
  • Use case: Best for ARM - based devices or low - memory environments.

Summary Table: Model Format Selection

📄 License

The model is licensed under the BigScience OpenRAIL - M v1 license agreement.

Model Stats

⚠️ Important Note

The Refact - 1.6B model was trained on text in English. Its performance on non - English languages is lower.

💡 Usage Tip

If you want to test the AI - Powered Network Monitor Assistant, click the chat icon (bottom right on any page), choose an AI assistant type (TurboLLM, FreeLLM, TestLLM), and try some example AI commands like "Give me info on my websites SSL certificate", "Check if my server is using quantum safe encyption for communication", "Run a quick Nmap vulnerability test".