Mistral 7B Instruct V0.3 AWQ

🚀 mistralai/Mistral-7B-Instruct-v0.3 AWQ

本项目是Mistral-7B-Instruct-v0.3模型的AWQ量化版本，能高效进行文本生成任务，在特定硬件和软件环境下可快速推理。

🚀 快速开始

安装必要的包

pip install --upgrade autoawq autoawq-kernels

Python代码示例

基础用法

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer, TextStreamer

model_path = "solidrust/Mistral-7B-Instruct-v0.3-AWQ"
system_message = "You are Mistral-7B-Instruct-v0.3, incarnated as a powerful AI. You were created by mistralai."

# Load model
model = AutoAWQForCausalLM.from_quantized(model_path,
                                          fuse_layers=True)
tokenizer = AutoTokenizer.from_pretrained(model_path,
                                          trust_remote_code=True)
streamer = TextStreamer(tokenizer,
                        skip_prompt=True,
                        skip_special_tokens=True)

# Convert prompt to tokens
prompt_template = """\
<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant"""

prompt = "You're standing on the surface of the Earth. "\
        "You walk one mile south, one mile west and one mile north. "\
        "You end up exactly where you started. Where are you?"

tokens = tokenizer(prompt_template.format(system_message=system_message,prompt=prompt),
                  return_tensors='pt').input_ids.cuda()

# Generate output
generation_output = model.generate(tokens,
                                  streamer=streamer,
                                  max_new_tokens=512)

✨ 主要特性

• 模型创建者：mistralai
• 原始模型：Mistral-7B-Instruct-v0.3
• 模型概要：Mistral-7B-Instruct-v0.3大语言模型（LLM）是Mistral-7B-v0.3的指令微调版本。与Mistral-7B-v0.2相比，Mistral-7B-v0.3有以下变化：
  • 词汇表扩展到32768
  • 支持v3分词器
  • 支持函数调用

📦 安装指南

安装必要的包，使用以下命令：

pip install --upgrade autoawq autoawq-kernels

💻 使用示例

基础用法

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer, TextStreamer

model_path = "solidrust/Mistral-7B-Instruct-v0.3-AWQ"
system_message = "You are Mistral-7B-Instruct-v0.3, incarnated as a powerful AI. You were created by mistralai."

# Load model
model = AutoAWQForCausalLM.from_quantized(model_path,
                                          fuse_layers=True)
tokenizer = AutoTokenizer.from_pretrained(model_path,
                                          trust_remote_code=True)
streamer = TextStreamer(tokenizer,
                        skip_prompt=True,
                        skip_special_tokens=True)

# Convert prompt to tokens
prompt_template = """\
<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant"""

prompt = "You're standing on the surface of the Earth. "\
        "You walk one mile south, one mile west and one mile north. "\
        "You end up exactly where you started. Where are you?"

tokens = tokenizer(prompt_template.format(system_message=system_message,prompt=prompt),
                  return_tensors='pt').input_ids.cuda()

# Generate output
generation_output = model.generate(tokens,
                                  streamer=streamer,
                                  max_new_tokens=512)

关于AWQ

AWQ是一种高效、准确且极快的低比特权重量化方法，目前支持4位量化。与GPTQ相比，在基于Transformer的推理中，它速度更快，并且在质量上与最常用的GPTQ设置相当或更好。

AWQ模型目前仅在Linux和Windows系统上支持，且仅支持NVIDIA GPU。macOS用户请使用GGUF模型。

它得到以下工具的支持：

• Text Generation Webui - 使用加载器：AutoAWQ
• vLLM - 版本0.2.2及更高版本支持所有模型类型
• Hugging Face Text Generation Inference (TGI)
• Transformers 版本4.35.0及更高版本，任何支持Transformers的代码或客户端
• AutoAWQ - 用于Python代码

📚 详细文档

📄 许可证

本模型使用的许可证为apache-2.0。