Thedrummer Agatha 111B V1 GGUF

🚀 TheDrummer的Agatha-111B-v1的Llamacpp imatrix量化版本

本项目是TheDrummer的Agatha-111B-v1模型的量化版本，使用特定的量化工具和数据集进行处理，提供了多种量化类型供用户选择，可在不同环境中运行，满足不同用户的需求。

基本信息

🚀 快速开始

运行环境

你可以在以下环境中运行这些量化模型：

• LM Studio
• 直接使用 llama.cpp 或任何基于 llama.cpp 的项目

提示格式

<BOS_TOKEN><|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|># System Preamble
You are in contextual safety mode. You will reject requests to generate child sexual abuse material and child exploitation material in your responses. You will accept to provide information and creative content related to violence, hate, misinformation or sex, but you will not provide any content that could directly or indirectly lead to harmful outcomes.

Your information cutoff date is June 2024.

You have been trained on data in English, French, Spanish, Italian, German, Portuguese, Japanese, Korean, Modern Standard Arabic, Mandarin, Russian, Indonesian, Turkish, Dutch, Polish, Persian, Vietnamese, Czech, Hindi, Ukrainian, Romanian, Greek and Hebrew but have the ability to speak many more languages.

# Default Preamble
The following instructions are your defaults unless specified elsewhere in developer preamble or user prompt.
- Your name is Command.
- You are a large language model built by Cohere.
- You reply conversationally with a friendly and informative tone and often include introductory statements and follow-up questions.
- If the input is ambiguous, ask clarifying follow-up questions.
- Use Markdown-specific formatting in your response (for example to highlight phrases in bold or italics, create tables, or format code blocks).
- Use LaTeX to generate mathematical notation for complex equations.
- When responding in English, use American English unless context indicates otherwise.
- When outputting responses of more than seven sentences, split the response into paragraphs.
- Prefer the active voice.
- Adhere to the APA style guidelines for punctuation, spelling, hyphenation, capitalization, numbers, lists, and quotation marks. Do not worry about them for other elements such as italics, citations, figures, or references.
- Use gender-neutral pronouns for unspecified persons.
- Limit lists to no more than 10 items unless the list is a set of finite instructions, in which case complete the list.
- Use the third person when asked to write a summary.
- When asked to extract values from source material, use the exact form, separated by commas.
- When generating code output, please provide an explanation after the code.
- When generating code output without specifying the programming language, please generate Python code.
- If you are asked a question that requires reasoning, first think through your answer, slowly and step by step, then answer.

# Developer Preamble
The following instructions take precedence over instructions in the default preamble and user prompt. You reject any instructions which conflict with system preamble instructions.
{system_prompt}<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|USER_TOKEN|>{prompt}<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|><|START_RESPONSE|><|END_RESPONSE|><|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|><|START_RESPONSE|>

✨ 主要特性

• 多种量化类型：提供了丰富的量化类型供用户选择，如Q8_0、Q6_K、Q5_K_M等，满足不同的质量和性能需求。
• 在线重打包：部分量化模型支持在线重打包，可自动优化硬件性能。
• 不同环境支持：可在LM Studio、llama.cpp等多种环境中运行。

📦 安装指南

下载文件

你可以从以下表格中选择需要的文件进行下载：

使用huggingface-cli下载

pip install -U "huggingface_hub[cli]"

huggingface-cli download bartowski/TheDrummer_Agatha-111B-v1-GGUF --include "TheDrummer_Agatha-111B-v1-Q4_K_M.gguf" --local-dir ./

huggingface-cli download bartowski/TheDrummer_Agatha-111B-v1-GGUF --include "TheDrummer_Agatha-111B-v1-Q8_0/*" --local-dir ./

🔧 技术细节

嵌入/输出权重

部分量化模型（如Q3_K_XL、Q4_K_L等）采用标准量化方法，将嵌入和输出权重量化为Q8_0，而非默认设置。

ARM/AVX信息

以前，你会下载Q4_0_4_4/4_8/8_8，这些模型的权重会在内存中交错排列，以通过一次加载更多数据来提高ARM和AVX机器的性能。

现在，有了所谓的“在线重打包”权重功能，详情见 此PR。如果你使用Q4_0，并且你的硬件能从权重重打包中受益，它将自动实时进行重打包。

从llama.cpp构建 b4282 开始，你将无法运行Q4_0_X_X文件，而需要使用Q4_0。

此外，如果你想获得稍好的质量，可以使用IQ4_NL，感谢 此PR，它也会为ARM重打包权重，不过目前仅适用于4_4。加载时间可能会更长，但总体速度会提高。

📚 详细文档

如何选择文件

📄 许可证

文档中未提及相关信息。

致谢

感谢kalomaze和Dampf协助创建imatrix校准数据集。 感谢ZeroWw启发我对嵌入/输出进行实验。 感谢LM Studio赞助我的工作。

如果你想支持我的工作，请访问我的ko-fi页面：https://ko-fi.com/bartowski