Bitnet B1.58 2B 4T GGUF

🚀 IQ2_BN and IQ2_BN_R4 for ik_llama.cpp

This project provides the IQ2_BN and IQ2_BN_R4 versions of microsoft/bitnet-b1.58-2B-4T-gguf for use with ik_llama.cpp. It offers a solution for text generation tasks with specific quantization settings.

✨ Features

• Quantized Model: Based on the quantized version of microsoft/bitnet-b1.58-2B-4T, which can potentially reduce memory usage and speed up inference.
• Runtime Conversion: You can use the -rtr option on the IQ2_BN version to convert it at runtime.
• Chat Template: Although the chat template in the model seems incorrect (inherited from the original Microsoft GGUF), correct usage examples are provided to enable multi - turn conversations.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

The correct usage example from the transformers PR is as follows:

<|begin_of_text|>User: Hey, are you conscious? Can you talk to me?<|eot_id|>Assistant:

This example can be used for multi - turn conversations.

Advanced Usage

The more general template (sourced from the paper) is:

<|begin_of_text|>System: {system_message}<|eot_id|>
User: {user_message_1}<|eot_id|>
Assistant: {assistant_message_1}<|eot_id|>
User: {user_message_2}<|eot_id|>
Assistant: {assistant_message_2}<|eot_id|>

📚 Documentation

The chat template in the model appears to be incorrect, but you can refer to the example from the transformers PR for correct usage.

📄 License

This project is licensed under the MIT license.

💡 Usage Tip I recommend using the IQ2_BN_R4 version. However, you can use -rtr on the IQ2_BN version to convert it at runtime.