Internlm3 8b Instruct Gguf

🚀 InternLM3-8B-Instruct GGUF模型

internlm3-8b-instruct的GGUF格式模型可借助llama.cpp这一广受欢迎的大语言模型（LLM）推理开源框架，在本地及云端等多种硬件平台上使用。本仓库提供了半精度和多种低比特量化版本（如q5_0、q5_k_m、q6_k和q8_0）的internlm3-8b-instruct GGUF格式模型。

接下来，我们将依次介绍安装步骤、模型下载方法，最后通过具体示例说明模型推理和服务部署的方式。

📦 安装指南

我们建议从源代码构建llama.cpp。以下是Linux CUDA平台的示例代码，其他平台的安装说明请参考官方指南。

• 步骤一：创建conda环境并安装cmake

conda create --name internlm3 python=3.10 -y
conda activate internlm3
pip install cmake

• 步骤二：克隆源代码并构建项目

git clone --depth=1 https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

所有构建的目标文件都可以在build/bin子目录中找到。

在后续内容中，我们假设工作目录为llama.cpp的根目录。

📥 下载模型

在简介部分中，我们提到本仓库包含了几种不同计算精度的模型。你可以根据自己的需求下载合适的模型。例如，可按以下方式下载internlm3-8b-instruct-fp16.gguf：

pip install huggingface-hub
huggingface-cli download internlm/internlm3-8b-instruct-gguf internlm3-8b-instruct.gguf --local-dir . --local-dir-use-symlinks False

💻 使用示例

基础用法

推理

你可以使用llama-cli进行推理。关于llama-cli的详细说明，请参考此指南。

聊天示例

以下是使用思维系统提示的示例：

thinking_system_prompt="<|im_start|>system\nYou are an expert mathematician with extensive experience in mathematical competitions. You approach problems through systematic thinking and rigorous reasoning. When solving problems, follow these thought processes:\n## Deep Understanding\nTake time to fully comprehend the problem before attempting a solution. Consider:\n- What is the real question being asked?\n- What are the given conditions and what do they tell us?\n- Are there any special restrictions or assumptions?\n- Which information is crucial and which is supplementary?\n## Multi-angle Analysis\nBefore solving, conduct thorough analysis:\n- What mathematical concepts and properties are involved?\n- Can you recall similar classic problems or solution methods?\n- Would diagrams or tables help visualize the problem?\n- Are there special cases that need separate consideration?\n## Systematic Thinking\nPlan your solution path:\n- Propose multiple possible approaches\n- Analyze the feasibility and merits of each method\n- Choose the most appropriate method and explain why\n- Break complex problems into smaller, manageable steps\n## Rigorous Proof\nDuring the solution process:\n- Provide solid justification for each step\n- Include detailed proofs for key conclusions\n- Pay attention to logical connections\n- Be vigilant about potential oversights\n## Repeated Verification\nAfter completing your solution:\n- Verify your results satisfy all conditions\n- Check for overlooked special cases\n- Consider if the solution can be optimized or simplified\n- Review your reasoning process\nRemember:\n1. Take time to think thoroughly rather than rushing to an answer\n2. Rigorously prove each key conclusion\n3. Keep an open mind and try different approaches\n4. Summarize valuable problem-solving methods\n5. Maintain healthy skepticism and verify multiple times\nYour response should reflect deep mathematical understanding and precise logical thinking, making your solution path and reasoning clear to others.\nWhen you're ready, present your complete solution with:\n- Clear problem understanding\n- Detailed solution process\n- Key insights\n- Thorough verification\nFocus on clear, logical progression of ideas and thorough explanation of your mathematical reasoning. Provide answers in the same language as the user asking the question, repeat the final answer using a '\\boxed{}' without any units, you have [[8192]] tokens to complete the answer.\n<|im_end|>\n"

build/bin/llama-cli \
    --model internlm3-8b-instruct.gguf  \
    --predict 2048 \
    --ctx-size 8192 \
    --gpu-layers 48 \
    --temp 0.8 \
    --top-p 0.8 \
    --top-k 50 \
    --seed 1024 \
    --color \
    --prompt "$thinking_system_prompt" \
    --interactive \
    --multiline-input \
    --conversation \
    --verbose \
    --logdir workdir/logdir \
    --in-prefix "<|im_start|>user\n" \
    --in-suffix "<|im_end|>\n<|im_start|>assistant\n"

然后输入你的问题，例如Given the function\(f(x)=\mathrm{e}^{x}-ax - a^{3}\),\n(1) When \(a = 1\), find the equation of the tangent line to the curve \(y = f(x)\) at the point \((1,f(1))\).\n(2) If \(f(x)\) has a local minimum and the minimum value is less than \(0\), determine the range of values for \(a\).

函数调用示例

llama-cli示例：

build/bin/llama-cli \
    --model internlm3-8b-instruct.gguf \
    --predict 512 \
    --ctx-size 4096 \
    --gpu-layers 48 \
    --temp 0.8 \
    --top-p 0.8 \
    --top-k 50 \
    --seed 1024 \
    --color \
    --prompt '<|im_start|>system\nYou are InternLM-Chat, a harmless AI assistant.<|im_end|>\n<|im_start|>system name=<|plugin|>[{"name": "get_current_weather", "parameters": {"required": ["location"], "type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "unit": {"type": "string"}}}, "description": "Get the current weather in a given location"}]<|im_end|>\n<|im_start|>user\n' \
    --interactive \
    --multiline-input \
    --conversation \
    --verbose \
    --in-suffix "<|im_end|>\n<|im_start|>assistant\n" \
    --special

对话结果：

<s><|im_start|>system
You are InternLM-Chat, a harmless AI assistant.<|im_end|>
<|im_start|>system name=<|plugin|>[{"name": "get_current_weather", "parameters": {"required": ["location"], "type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "unit": {"type": "string"}}}, "description": "Get the current weather in a given location"}]<|im_end|>
<|im_start|>user

> I want to know today's weather in Shanghai
I need to use the get_current_weather function to get the current weather in Shanghai.<|action_start|><|plugin|>
{"name": "get_current_weather", "parameters": {"location": "Shanghai"}}<|action_end|>32
<|im_end|>

> <|im_start|>environment name=<|plugin|>\n{"temperature": 22}
The current temperature in Shanghai is 22 degrees Celsius.<|im_end|>

> 

服务部署

llama.cpp提供了一个兼容OpenAI API的服务器llama-server。你可以按以下方式将internlm3-8b-instruct.gguf部署为服务：

./build/bin/llama-server -m ./internlm3-8b-instruct.gguf -ngl 48

在客户端，你可以通过OpenAI API访问该服务：

from openai import OpenAI
client = OpenAI(
    api_key='YOUR_API_KEY',
    base_url='http://localhost:8080/v1'
)
model_name = client.models.list().data[0].id
response = client.chat.completions.create(
  model=model_name,
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": " provide three suggestions about time management"},
  ],
  temperature=0.8,
  top_p=0.8
)
print(response)

📄 许可证

本项目采用Apache-2.0许可证。