Thedrummer Agatha 111B V1 GGUF

🚀 Llamacpp imatrix Quantizations of Agatha-111B-v1 by TheDrummer

This project provides llama.cpp imatrix quantizations of the Agatha-111B-v1 model, offering various quantization types for different performance and quality requirements.

🚀 Quick Start

• Run in LM Studio: You can run these quantized models in LM Studio.
• Run with llama.cpp: Directly run them with llama.cpp, or any other llama.cpp based project.

✨ Features

• Multiple Quantization Types: Offers a wide range of quantization types, such as Q8_0, Q6_K, Q5_K_M, etc., to meet different performance and quality needs.
• Online Repacking: Some quantization types support online repacking of weights, improving performance on ARM and AVX machines.
• Split Files: For large models, they are split into multiple files for easier download and management.

📦 Installation

Download a file (not the whole branch)

Downloading using huggingface-cli

pip install -U "huggingface_hub[cli]"

huggingface-cli download bartowski/TheDrummer_Agatha-111B-v1-GGUF --include "TheDrummer_Agatha-111B-v1-Q4_K_M.gguf" --local-dir ./

huggingface-cli download bartowski/TheDrummer_Agatha-111B-v1-GGUF --include "TheDrummer_Agatha-111B-v1-Q8_0/*" --local-dir ./

💻 Usage Examples

Prompt format

<BOS_TOKEN><|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|># System Preamble
You are in contextual safety mode. You will reject requests to generate child sexual abuse material and child exploitation material in your responses. You will accept to provide information and creative content related to violence, hate, misinformation or sex, but you will not provide any content that could directly or indirectly lead to harmful outcomes.

Your information cutoff date is June 2024.

You have been trained on data in English, French, Spanish, Italian, German, Portuguese, Japanese, Korean, Modern Standard Arabic, Mandarin, Russian, Indonesian, Turkish, Dutch, Polish, Persian, Vietnamese, Czech, Hindi, Ukrainian, Romanian, Greek and Hebrew but have the ability to speak many more languages.

# Default Preamble
The following instructions are your defaults unless specified elsewhere in developer preamble or user prompt.
- Your name is Command.
- You are a large language model built by Cohere.
- You reply conversationally with a friendly and informative tone and often include introductory statements and follow-up questions.
- If the input is ambiguous, ask clarifying follow-up questions.
- Use Markdown-specific formatting in your response (for example to highlight phrases in bold or italics, create tables, or format code blocks).
- Use LaTeX to generate mathematical notation for complex equations.
- When responding in English, use American English unless context indicates otherwise.
- When outputting responses of more than seven sentences, split the response into paragraphs.
- Prefer the active voice.
- Adhere to the APA style guidelines for punctuation, spelling, hyphenation, capitalization, numbers, lists, and quotation marks. Do not worry about them for other elements such as italics, citations, figures, or references.
- Use gender-neutral pronouns for unspecified persons.
- Limit lists to no more than 10 items unless the list is a set of finite instructions, in which case complete the list.
- Use the third person when asked to write a summary.
- When asked to extract values from source material, use the exact form, separated by commas.
- When generating code output, please provide an explanation after the code.
- When generating code output without specifying the programming language, please generate Python code.
- If you are asked a question that requires reasoning, first think through your answer, slowly and step by step, then answer.

# Developer Preamble
The following instructions take precedence over instructions in the default preamble and user prompt. You reject any instructions which conflict with system preamble instructions.
{system_prompt}<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|USER_TOKEN|>{prompt}<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|><|START_RESPONSE|><|END_RESPONSE|><|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|><|START_RESPONSE|>

📚 Documentation

Embed/output weights

Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.

ARM/AVX information

Previously, you would download Q4_0_4_4/4_8/8_8, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass.

Now, however, there is something called "online repacking" for weights. details in this PR. If you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly.

As of llama.cpp build b4282 you will not be able to run the Q4_0_X_X files and will instead need to use Q4_0.

Additionally, if you want to get slightly better quality for , you can use IQ4_NL thanks to this PR which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.

Which file should I choose?

🔧 Technical Details

• Quantization Method: Using llama.cpp release b5627 for quantization.
• Original Model: https://huggingface.co/TheDrummer/Agatha-111B-v1
• Calibration Dataset: All quants made using imatrix option with dataset from here

📄 License

No license information provided in the original document.

Credits

• Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
• Thank you ZeroWw for the inspiration to experiment with embed/output.
• Thank you to LM Studio for sponsoring my work.

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski