Bitnet B1.58 2B 4T Bf16

🚀 BitNet b1.58 2B4T - Scaling Native 1-bit LLM

This repository holds the weights for BitNet b1.58 2B4T, the world's first open - source, native 1 - bit Large Language Model (LLM) with around 2 billion parameters, developed by Microsoft Research. Trained on a 4 - trillion - token corpus, it shows that native 1 - bit LLMs can match the performance of leading open - weight, full - precision models of similar size while offering significant computational efficiency advantages in memory, energy, and latency.

➡️ Technical Report: BitNet b1.58 2B4T Technical Report

➡️ Official Inference Code: microsoft/BitNet (bitnet.cpp)

✨ Features

• High Efficiency: Native 1.58 - bit weights and 8 - bit activations (W1.58A8) bring substantial computational efficiency improvements.
• Good Performance: Achieves comparable performance to leading open - weight, full - precision models of similar size.
• Multiple Training Stages: Undergoes pre - training, supervised fine - tuning (SFT), and direct preference optimization (DPO) to align with human preferences.

📦 Installation

Requirements

pip install git+https://github.com/huggingface/transformers.git@096f25ae1f501a084d8ff2dcaf25fbc2bd60eba4

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "microsoft/bitnet-b1.58-2B-4T"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16
)

# Apply the chat template
messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "How are you?"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
chat_input = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate response
chat_outputs = model.generate(**chat_input, max_new_tokens=50)
response = tokenizer.decode(chat_outputs[0][chat_input['input_ids'].shape[-1]:], skip_special_tokens=True) # Decode only the response part
print("\nAssistant Response:", response)

Advanced Usage

Please refer to the bitnet.cpp GitHub repository for detailed compilation steps, usage examples, and command - line options.

📚 Documentation

Model Variants

Several versions of the model weights are available on Hugging Face:

• microsoft/bitnet-b1.58-2B-4T: Contains the packed 1.58 - bit weights optimized for efficient inference. Use this for deployment.
• microsoft/bitnet-b1.58-2B-4T-bf16 (This repository): Contains the master weights in BF16 format. Use this only for training or fine - tuning purposes.
• microsoft/bitnet-b1.58-2B-4T-gguf: Contains the model weights in GGUF format, compatible with the bitnet.cpp library for CPU inference.

Model Details

Evaluation

BitNet b1.58 2B4T was evaluated against leading open - weight full - precision LLMs of similar size. Below are the key results (all models are instruction - tuned versions):

*LLaMA 3.2 1B uses pruning & distillation.

**Gemma - 3 1B uses distillation.

🔧 Technical Details

• Quantization Scheme: The model uses native 1.58 - bit weights and 8 - bit activations (W1.58A8). The weights are quantized to ternary values {-1, 0, +1} using absmean quantization during the forward pass, and activations are quantized to 8 - bit integers using absmax quantization (per - token). It is crucial that the model was trained from scratch with this quantization scheme, not post - training quantized.
• Training Process: It goes through three training stages: pre - training, supervised fine - tuning (SFT), and direct preference optimization (DPO). Each stage has its own specific training data, loss functions, and hyperparameter settings.

📄 License

The model weights and code are released under the MIT License.

⚠️ Important Note

Please do NOT expect performance efficiency gains (in terms of speed, latency, or energy consumption) when using this model with the standard transformers library, even with the required fork. The current execution paths within transformers do not contain the specialized, highly optimized computational kernels required to leverage the advantages of the BitNet architecture. Running the model via transformers will likely result in inference speeds and energy usage comparable to, or potentially worse than, standard full - precision models within this framework on both CPU and GPU. While you might observe reduced memory usage due to the quantized weights, the primary computational efficiency benefits are not accessible through this standard transformers usage path. For achieving the efficiency benefits demonstrated in the technical paper, you MUST use the dedicated C++ implementation: bitnet.cpp.

💡 Usage Tip

This model is intended for research and development purposes. While efforts have been made to align it using SFT and DPO, it may still produce outputs that are unexpected, biased, or inaccurate. Please use responsibly.