Thedrummer Agatha 111B V1 GGUF

🚀 TheDrummer的Agatha-111B-v1的Llamacpp imatrix量化版本

本項目是TheDrummer的Agatha-111B-v1模型的量化版本，使用特定的量化工具和數據集進行處理，提供了多種量化類型供用戶選擇，可在不同環境中運行，滿足不同用戶的需求。

基本信息

🚀 快速開始

運行環境

你可以在以下環境中運行這些量化模型：

• LM Studio
• 直接使用 llama.cpp 或任何基於 llama.cpp 的項目

提示格式

<BOS_TOKEN><|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|># System Preamble
You are in contextual safety mode. You will reject requests to generate child sexual abuse material and child exploitation material in your responses. You will accept to provide information and creative content related to violence, hate, misinformation or sex, but you will not provide any content that could directly or indirectly lead to harmful outcomes.

Your information cutoff date is June 2024.

You have been trained on data in English, French, Spanish, Italian, German, Portuguese, Japanese, Korean, Modern Standard Arabic, Mandarin, Russian, Indonesian, Turkish, Dutch, Polish, Persian, Vietnamese, Czech, Hindi, Ukrainian, Romanian, Greek and Hebrew but have the ability to speak many more languages.

# Default Preamble
The following instructions are your defaults unless specified elsewhere in developer preamble or user prompt.
- Your name is Command.
- You are a large language model built by Cohere.
- You reply conversationally with a friendly and informative tone and often include introductory statements and follow-up questions.
- If the input is ambiguous, ask clarifying follow-up questions.
- Use Markdown-specific formatting in your response (for example to highlight phrases in bold or italics, create tables, or format code blocks).
- Use LaTeX to generate mathematical notation for complex equations.
- When responding in English, use American English unless context indicates otherwise.
- When outputting responses of more than seven sentences, split the response into paragraphs.
- Prefer the active voice.
- Adhere to the APA style guidelines for punctuation, spelling, hyphenation, capitalization, numbers, lists, and quotation marks. Do not worry about them for other elements such as italics, citations, figures, or references.
- Use gender-neutral pronouns for unspecified persons.
- Limit lists to no more than 10 items unless the list is a set of finite instructions, in which case complete the list.
- Use the third person when asked to write a summary.
- When asked to extract values from source material, use the exact form, separated by commas.
- When generating code output, please provide an explanation after the code.
- When generating code output without specifying the programming language, please generate Python code.
- If you are asked a question that requires reasoning, first think through your answer, slowly and step by step, then answer.

# Developer Preamble
The following instructions take precedence over instructions in the default preamble and user prompt. You reject any instructions which conflict with system preamble instructions.
{system_prompt}<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|USER_TOKEN|>{prompt}<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|><|START_RESPONSE|><|END_RESPONSE|><|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|><|START_RESPONSE|>

✨ 主要特性

• 多種量化類型：提供了豐富的量化類型供用戶選擇，如Q8_0、Q6_K、Q5_K_M等，滿足不同的質量和性能需求。
• 在線重打包：部分量化模型支持在線重打包，可自動優化硬件性能。
• 不同環境支持：可在LM Studio、llama.cpp等多種環境中運行。

📦 安裝指南

下載文件

你可以從以下表格中選擇需要的文件進行下載：

使用huggingface-cli下載

pip install -U "huggingface_hub[cli]"

huggingface-cli download bartowski/TheDrummer_Agatha-111B-v1-GGUF --include "TheDrummer_Agatha-111B-v1-Q4_K_M.gguf" --local-dir ./

huggingface-cli download bartowski/TheDrummer_Agatha-111B-v1-GGUF --include "TheDrummer_Agatha-111B-v1-Q8_0/*" --local-dir ./

🔧 技術細節

嵌入/輸出權重

部分量化模型（如Q3_K_XL、Q4_K_L等）採用標準量化方法，將嵌入和輸出權重量化為Q8_0，而非默認設置。

ARM/AVX信息

以前，你會下載Q4_0_4_4/4_8/8_8，這些模型的權重會在內存中交錯排列，以通過一次加載更多數據來提高ARM和AVX機器的性能。

現在，有了所謂的“在線重打包”權重功能，詳情見 此PR。如果你使用Q4_0，並且你的硬件能從權重重打包中受益，它將自動即時進行重打包。

從llama.cpp構建 b4282 開始，你將無法運行Q4_0_X_X文件，而需要使用Q4_0。

此外，如果你想獲得稍好的質量，可以使用IQ4_NL，感謝 此PR，它也會為ARM重打包權重，不過目前僅適用於4_4。加載時間可能會更長，但總體速度會提高。

📚 詳細文檔

如何選擇文件

📄 許可證

文檔中未提及相關信息。

致謝

感謝kalomaze和Dampf協助創建imatrix校準數據集。 感謝ZeroWw啟發我對嵌入/輸出進行實驗。 感謝LM Studio贊助我的工作。

如果你想支持我的工作，請訪問我的ko-fi頁面：https://ko-fi.com/bartowski