Mistral 7B Instruct V0.3 AWQ

🚀 mistralai/Mistral-7B-Instruct-v0.3 AWQ

本項目是Mistral-7B-Instruct-v0.3模型的AWQ量化版本，能高效進行文本生成任務，在特定硬件和軟件環境下可快速推理。

🚀 快速開始

安裝必要的包

pip install --upgrade autoawq autoawq-kernels

Python代碼示例

基礎用法

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer, TextStreamer

model_path = "solidrust/Mistral-7B-Instruct-v0.3-AWQ"
system_message = "You are Mistral-7B-Instruct-v0.3, incarnated as a powerful AI. You were created by mistralai."

# Load model
model = AutoAWQForCausalLM.from_quantized(model_path,
                                          fuse_layers=True)
tokenizer = AutoTokenizer.from_pretrained(model_path,
                                          trust_remote_code=True)
streamer = TextStreamer(tokenizer,
                        skip_prompt=True,
                        skip_special_tokens=True)

# Convert prompt to tokens
prompt_template = """\
<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant"""

prompt = "You're standing on the surface of the Earth. "\
        "You walk one mile south, one mile west and one mile north. "\
        "You end up exactly where you started. Where are you?"

tokens = tokenizer(prompt_template.format(system_message=system_message,prompt=prompt),
                  return_tensors='pt').input_ids.cuda()

# Generate output
generation_output = model.generate(tokens,
                                  streamer=streamer,
                                  max_new_tokens=512)

✨ 主要特性

• 模型創建者：mistralai
• 原始模型：Mistral-7B-Instruct-v0.3
• 模型概要：Mistral-7B-Instruct-v0.3大語言模型（LLM）是Mistral-7B-v0.3的指令微調版本。與Mistral-7B-v0.2相比，Mistral-7B-v0.3有以下變化：
  • 詞彙表擴展到32768
  • 支持v3分詞器
  • 支持函數調用

📦 安裝指南

安裝必要的包，使用以下命令：

pip install --upgrade autoawq autoawq-kernels

💻 使用示例

基礎用法

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer, TextStreamer

model_path = "solidrust/Mistral-7B-Instruct-v0.3-AWQ"
system_message = "You are Mistral-7B-Instruct-v0.3, incarnated as a powerful AI. You were created by mistralai."

# Load model
model = AutoAWQForCausalLM.from_quantized(model_path,
                                          fuse_layers=True)
tokenizer = AutoTokenizer.from_pretrained(model_path,
                                          trust_remote_code=True)
streamer = TextStreamer(tokenizer,
                        skip_prompt=True,
                        skip_special_tokens=True)

# Convert prompt to tokens
prompt_template = """\
<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant"""

prompt = "You're standing on the surface of the Earth. "\
        "You walk one mile south, one mile west and one mile north. "\
        "You end up exactly where you started. Where are you?"

tokens = tokenizer(prompt_template.format(system_message=system_message,prompt=prompt),
                  return_tensors='pt').input_ids.cuda()

# Generate output
generation_output = model.generate(tokens,
                                  streamer=streamer,
                                  max_new_tokens=512)

關於AWQ

AWQ是一種高效、準確且極快的低比特權重量化方法，目前支持4位量化。與GPTQ相比，在基於Transformer的推理中，它速度更快，並且在質量上與最常用的GPTQ設置相當或更好。

AWQ模型目前僅在Linux和Windows系統上支持，且僅支持NVIDIA GPU。macOS用戶請使用GGUF模型。

它得到以下工具的支持：

• Text Generation Webui - 使用加載器：AutoAWQ
• vLLM - 版本0.2.2及更高版本支持所有模型類型
• Hugging Face Text Generation Inference (TGI)
• Transformers 版本4.35.0及更高版本，任何支持Transformers的代碼或客戶端
• AutoAWQ - 用於Python代碼

📚 詳細文檔

📄 許可證

本模型使用的許可證為apache-2.0。