Llama3 8B Chinese Chat GGUF F16

🚀 Llama3-8B-Chinese-Chat

Llama3-8B-Chinese-Chat is an instruction-tuned language model for Chinese & English users with various abilities such as roleplaying & tool-using built upon the Meta-Llama-3-8B-Instruct model.

🚀 Quick Start

🌟 All instructions on how to download, use, and reproduce our various kinds of models are included at this GitHub repo. If you like our models, we would greatly appreciate it if you could star our Github repository. Additionally, please click "like" on our HuggingFace repositories. Thank you!

❗️❗️❗️NOTICE: The main branch contains the f16 GGUF files for Llama3-8B-Chinese-Chat-v2.1. If you want to use our f16 GGUF files for Llama3-8B-Chinese-Chat-v1, please refer to the v1 branch; if you want to use our f16 GGUF files for Llama3-8B-Chinese-Chat-v2, please refer to the v2 branch.

❗️❗️❗️NOTICE: For optimal performance, we refrain from fine-tuning the model's identity. Thus, inquiries such as "Who are you" or "Who developed you" may yield random responses that are not necessarily accurate.

✨ Features

Llama3-8B-Chinese-Chat is an instruction-tuned language model for Chinese & English users with various abilities such as roleplaying & tool-using built upon the Meta-Llama-3-8B-Instruct model.

📦 Installation

Reproduce the model

To reproduce Llama3-8B-Chinese-Chat-v2 (to reproduce Llama3-8B-Chinese-Chat-v1, please refer to this link):

git clone https://github.com/hiyouga/LLaMA-Factory.git
git reset --hard 32347901d4af94ccd72b3c7e1afaaceb5cb3d26a    # For Llama3-8B-Chinese-Chat-v1: 836ca0558698206bbf4e3b92533ad9f67c9f9864

cd LLaMA-Factory

# Our dataset used for Llama3-8B-Chinese-Chat-v2 will be released soon. If you want to reproduce Llama3-8B-Chinese-Chat-v1, you can set `Your_Dataset_Name_or_PATH=dpo_mix_en,dpo_mix_zh`.
deepspeed --num_gpus 8 src/train_bash.py \
    --deepspeed ${Your_Deepspeed_Config_Path} \
    --stage orpo \
    --do_train \
    --model_name_or_path meta-llama/Meta-Llama-3-8B-Instruct \
    --dataset ${Your_Dataset_Name_or_PATH} \
    --template llama3 \
    --finetuning_type full \
    --output_dir ${Your_Output_Path} \
    --per_device_train_batch_size 1 \
    --per_device_eval_batch_size 1 \
    --gradient_accumulation_steps 8 \
    --lr_scheduler_type cosine \
    --log_level info \
    --logging_steps 5 \
    --save_strategy epoch \
    --save_total_limit 3 \
    --save_steps 100 \
    --learning_rate 5e-6 \
    --num_train_epochs 3.0 \
    --plot_loss \
    --do_eval false \
    --max_steps -1 \
    --bf16 true \
    --seed 42 \
    --warmup_ratio 0.1 \
    --cutoff_len 8192 \
    --flash_attn true \
    --orpo_beta 0.05 \
    --optim paged_adamw_32bit

💻 Usage Examples

Basic Usage

from llama_cpp import Llama

model = Llama(
    "/Your/Path/To/GGUF/File",
    verbose=False,
    n_gpu_layers=-1,
)

system_prompt = "You are a helpful assistant."

def generate_reponse(_model, _messages, _max_tokens=8192):
    _output = _model.create_chat_completion(
        _messages,
        stop=["<|eot_id|>", "<|end_of_text|>"],
        max_tokens=_max_tokens,
    )["choices"][0]["message"]["content"]
    return _output

# The following are some examples

messages = [
    {
        "role": "system",
        "content": system_prompt,
    },
    {"role": "user", "content": "写一首诗吧"},
]

print(generate_reponse(model, messages))

📚 Documentation

Model Summary

Introduction

This is the first model specifically fine-tuned for Chinese & English user through ORPO [1] based on the Meta-Llama-3-8B-Instruct model.

Compared to the original Meta-Llama-3-8B-Instruct model, our Llama3-8B-Chinese-Chat-v1 model significantly reduces the issues of "Chinese questions with English answers" and the mixing of Chinese and English in responses.

Compared to Llama3-8B-Chinese-Chat-v1, our Llama3-8B-Chinese-Chat-v2 model significantly increases the training data size (from 20K to 100K), which introduces great performance enhancement, especially in roleplay, tool using, and math.

[1] Hong, Jiwoo, Noah Lee, and James Thorne. "Reference-free Monolithic Preference Optimization with Odds Ratio." arXiv preprint arXiv:2403.07691 (2024).

Training framework: LLaMA-Factory.

Training details:

• epochs: 2
• learning rate: 5e-6
• learning rate scheduler type: cosine
• Warmup ratio: 0.1
• cutoff len (i.e. context length): 8192
• orpo beta (i.e. $\lambda$ in the ORPO paper): 0.05
• global batch size: 128
• fine-tuning type: full parameters
• optimizer: paged_adamw_32bit

Updates

• 🚀🚀🚀 [May 6, 2024] We now introduce Llama3-8B-Chinese-Chat-v2.1! Compared to v1, the training dataset of v2.1 is 5x larger (~100K preference pairs), and it exhibits significant enhancements, especially in roleplay, function calling, and math capabilities! Compared to v2, v2.1 surpasses v2 in math and is less prone to including English words in Chinese responses. The training dataset of Llama3-8B-Chinese-Chat-v2.1 will be released soon. If you love our Llama3-8B-Chinese-Chat-v1 or v2, you won't want to miss out on Llama3-8B-Chinese-Chat-v2.1!
• 🔥 We provide an online interactive demo for Llama3-8B-Chinese-Chat-v2 here. Have fun with our latest model!
• 🔥 We provide the official Ollama model for the q4_0 GGUF version of Llama3-8B-Chinese-Chat-v2.1 at wangshenzhi/llama3-8b-chinese-chat-ollama-q4! Run the following command for quick use of this model: ollama run wangshenzhi/llama3-8b-chinese-chat-ollama-q4.
• 🔥 We provide the official Ollama model for the q8_0 GGUF version of Llama3-8B-Chinese-Chat-v2.1 at wangshenzhi/llama3-8b-chinese-chat-ollama-q8! Run the following command for quick use of this model: ollama run wangshenzhi/llama3-8b-chinese-chat-ollama-q8.
• 🔥 We provide the official Ollama model for the f16 GGUF version of Llama3-8B-Chinese-Chat-v2.1 at wangshenzhi/llama3-8b-chinese-chat-ollama-fp16! Run the following command for quick use of this model: ollama run wangshenzhi/llama3-8b-chinese-chat-ollama-fp16.
• 🔥 We provide the official q4_0 GGUF version of Llama3-8B-Chinese-Chat-v2.1 at https://huggingface.co/shenzhi-wang/Llama3-8B-Chinese-Chat-GGUF-4bit!
• 🔥 We provide the official q8_0 GGUF version of Llama3-8B-Chinese-Chat-v2.1 at https://huggingface.co/shenzhi-wang/Llama3-8B-Chinese-Chat-GGUF-8bit!
• 🔥 We provide the official f16 GGUF version of Llama3-8B-Chinese-Chat-v2.1 at https://huggingface.co/shenzhi-wang/Llama3-8B-Chinese-Chat-GGUF-f16!

Examples

The following are some examples generated by Llama3-8B-Chinese-Chat-v2.1, including examples of role playing, function calling, math, RuoZhiBa (弱智吧), safety, writing, and coding, etc.

For the examples generated by Llama3-8B-Chinese-Chat-v1, please refer to this link.

For the examples generated by Llama3-8B-Chinese-Chat-v2, please refer to this link.

Roleplay Taylor Swift, and use beautiful lyrics to answer questions. Do you understand?

How are you today?

What do you want to say to your fans?

扮演周杰伦，用rap的方式回答问题。 

你的新歌准备得怎么样了？

你想对你的粉丝说些什么？

Roleplay Shakespeare. Do you understand?