Bielik 4.5B V3.0 Instruct FP8 Dynamic

🚀 Bielik-4.5B-v3.0-Instruct-FP8-Dynamic

This model is obtained by quantizing the weights and activations of Bielik-4.5B-v3.0-Instruct to FP8 data type, enabling efficient inference with vLLM >= 0.5.0 or SGLang.

🚀 Quick Start

This model was obtained by quantizing the weights and activations of Bielik-4.5B-v3.0-Instruct to FP8 data type, ready for inference with vLLM >= 0.5.0 or SGLang. AutoFP8 is used for quantization. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Only the weights and activations of the linear operators within transformers blocks are quantized. Symmetric per-tensor quantization is applied, in which a single linear scaling maps the FP8 representations of the quantized weights and activations.

🔗 Technical report: https://arxiv.org/abs/2505.02550

FP8 compuation is supported on Nvidia GPUs with compute capability > 8.9 (Ada Lovelace, Hopper).

⚠️ Important Note

Be aware that quantised models show reduced response quality and possible hallucinations!

✨ Features

Use with vLLM

This model can be deployed efficiently using the vLLM backend, as shown in the example below.

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

model_id = "speakleash/Bielik-4.5B-v3.0-Instruct-FP8-Dynamic"

sampling_params = SamplingParams(temperature=0.2, top_p=0.95, max_tokens=4096)

tokenizer = AutoTokenizer.from_pretrained(model_id)

messages = [
    {"role": "system", "content": "Jeste≈õ pomocnym asystentem Bielik."},
    {"role": "user", "content": "Kim był Mikołaj Kopernik i z czego zasłynął?"},
]

prompts = tokenizer.apply_chat_template(messages, tokenize=False)

llm = LLM(model=model_id, max_model_len=4096)

outputs = llm.generate(prompts, sampling_params)

generated_text = outputs[0].outputs[0].text
print(generated_text)

vLLM aslo supports OpenAI-compatible serving. See the documentation for more details.

Use with SGLang Runtime

Launch a server of SGLang Runtime:

python -m sglang.launch_server --model-path speakleash/Bielik-4.5B-v3.0-Instruct-FP8-Dynamic --port 30000

Then you can send http request or use OpenAI Compatible API.

import openai
client = openai.Client(
    base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="default",
    messages=[
        {"role": "system", "content": "Jeste≈õ pomocnym asystentem Bielik."},
        {"role": "user", "content": "Kim był Mikołaj Kopernik i z czego zasłynął?"},
    ],
    temperature=0,
    max_tokens=4096,
)
print(response)

📚 Documentation

Model description:

Responsible for model quantization

• Remigiusz Kinas^SpeakLeash - team leadership, conceptualizing, calibration data preparation, process creation and quantized model delivery.

📄 License

This model is under the Apache 2.0 license and Terms of Use.

📞 Contact Us

If you have any questions or suggestions, please use the discussion tab. If you want to contact us directly, join our Discord SpeakLeash.