BitNet b1.58 Open-source Large Language Model - Reduce computational cost while maintaining performance close to full precision!

Bitnet B1 58 Xl Q8 0 Gguf

Developed by BoscoTheDog

BitNet b1.58 is a large language model with 1.58-bit quantization. It reduces the computational resource requirements by lowering the weight precision while maintaining performance close to that of a full-precision model.

Large Language Model

Transformers

Open Source License:MIT #1.58-bit quantization #Efficient language model #Zero-shot learning

Downloads 326

Release Time : 6/23/2024

Model Overview

This model is a reproduction of the BitNet b1.58 paper. It was trained on 100B tokens using the RedPajama dataset, achieving an efficient 1.58-bit quantized LLM.

Model Features

1.58-bit quantization

Adopts an innovative 1.58-bit quantization technology, significantly reducing model storage and computational requirements

Efficient training

Optimizes the training process using a two-stage learning rate and weight decay strategy

Open-source model

All trained model parameters are fully open-source

Close to full-precision performance

Can still maintain performance close to that of an FP16 precision model under quantization

Model Capabilities

Text generation

Zero-shot learning

Language understanding

Question-answering tasks

Use Cases

Natural language processing

Open-domain question answering

Answer open-ended questions in various fields

Performs well in benchmarks such as ARC and HellaSwag

Text generation

Generate coherent and meaningful text

The perplexity (PPL) is close to that of a full-precision model

Research applications

Efficient LLM research

Study the impact of low-bit quantization on LLM performance

Provides a reference for the development of efficient LLMs

Models	PPL	ARCe	ARCc	HS	BQ	OQ	PQ	WGe	Avg
FP16 700M (reported)	12.33	54.7	23.0	37.0	60.0	20.2	68.9	54.8	45.5
BitNet b1.58 700M (reported)	12.87	51.8	21.4	35.1	58.2	20.0	68.1	55.2	44.3
BitNet b1.58 700M (reproduced)	12.78	51.4	21.8	35.0	59.6	20.6	67.5	55.4	44.5
FP16 1.3B (reported)	11.25	56.9	23.5	38.5	59.1	21.6	70.0	53.9	46.2
BitNet b1.58 1.3B (reported)	11.29	54.9	24.2	37.7	56.7	19.6	68.8	55.8	45.4
BitNet b1.58 1.3B (reproduced)	11.19	55.8	23.7	37.6	59.0	20.2	69.2	56.0	45.9
FP16 3B (reported)	10.04	62.1	25.6	43.3	61.8	24.6	72.1	58.2	49.7
BitNet b1.58 3B (reported)	9.91	61.4	28.3	42.9	61.5	26.6	71.5	59.3	50.2
BitNet b1.58 3B (reproduced)	9.88	60.9	28.0	42.3	58.3	26.0	71.4	60.3	49.6

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご

Bitnet B1 58 Xl Q8 0 Gguf

Model Overview

Model Features

Model Capabilities

Use Cases

🚀 BitNet b1.58 Paper Reproduction

📊 Results

🧪 Evaluation

📄 License