34b Beta

🚀 CausalLM 34B β

CausalLM 34B β is a powerful language model. It provides a demo on Hugging Face and has specific prompt formats. However, there are some precision issues with the model weights that will be fixed in the next update.

🚀 Quick Start

The demo of CausalLM 34B β can be accessed via the following link:

✨ Features

Prompt Format

The model uses the chatml prompt format.

Model Issues and Precautions

• Precision Issues: There are some issues with the model weights in terms of precision. In the next version update, we will roll back some progress and retrain to fix these issues as soon as possible.
• Inference Framework: Please do not use "accelerated inference frameworks" like VLLM temporarily. Instead, use Transformers for inference. Otherwise, due to precision issues, the output quality will be significantly degraded. If you need faster inference, you can consider using the q8_0 quantization (faster and better than bf16 vllm for this model only) with llama.cpp temporarily or wait for the official version.
• Repetition Penalty: no repetition_penalty!
• Quantization Calibration: Please do not use wikitext for quantization calibration because all wikitext have been re - aligned on synthetic dataset, and its distribution differs significantly from the original wikitext.

MT - Bench Score

The model achieves a score of 8.5 on MT - Bench. 

Contamination Detection

The data is from https://huggingface.co/spaces/Yeyito/llm_contamination_detector. It should be safe. It was not trained on the benchmark, but the contamination of the training dataset is unavoidable due to cost constraints.

📄 License

This project is licensed under the GPL - 3.0 license.