C4ai Command R Plus Imat.gguf

🚀 C4AI Command R+ GGUF Quantization

C4AI Command R+ is an open weights research release of a 104B billion parameter model. It offers highly advanced capabilities, such as Retrieval Augmented Generation (RAG) and tool use for automating sophisticated tasks. The model supports multi - step tool use, enabling it to combine multiple tools over multiple steps to accomplish difficult tasks. It's a multilingual model evaluated in 10 languages, optimized for reasoning, summarization, and question - answering.

📄 License

This model is licensed under CC - BY - NC - 4.0.

📋 Model Information

📅 Release Notes

2024 - 05 - 05

With commit 889bdd7 merged, we now have BPE pre - tokenization for this model, so all the quants will be refreshed.

2024 - 04 - 09

Support for this model has been merged into the main branch.

• Pull request PR #6491
• Commit 5dc9dd71

Noeda's fork will not work with these weights. You will need the main branch of llama.cpp.

⚠️ Important Note

Do not concatenate splits (or chunks). You need to use gguf - split to merge files if necessary (most likely not needed for most use cases).

✨ Features

• GGUF importance matrix (imatrix) quants for https://huggingface.co/CohereForAI/c4ai - command - r - plus.
• The importance matrix is trained for ~100K tokens (200 batches of 512 tokens) using wiki.train.raw.
• Which GGUF is right for me? (from Artefact2) - The X - axis is file size and the Y - axis is perplexity (lower perplexity means better quality). Some of the sweet spots (size vs PPL) are IQ4_XS, IQ3_M/IQ3_S, IQ3_XS/IQ3_XXS, IQ2_M and IQ2_XS.
• The imatrix is being used on the K - quants as well (only for < Q6_K).
• It's not necessary, but you could merge GGUFs with gguf - split --merge <first - chunk> <output - file>. This is not required since f482bb2e.
• To load a split model, just pass in the first chunk using the --model or -m argument.
• What is the importance matrix (imatrix)? You can read more about it from the author here. Some other info [here](https://huggingface.co/dranger003/c4ai - command - r - plus - iMat.GGUF/discussions/2#6612840b8377af8668066682).
• How do I use imatrix quants? Just like any other GGUF, the .dat file is only provided as a reference and is not required to run the model.
• If your last resort is to use an IQ1 quant, then go for IQ1_M.
• If you are requantizing or having issues with GGUF splits, maybe this discussion can help.

📊 Model Parameters

Layer and Context Information

64

131072

<BOS_TOKEN><|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>{system}<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|USER_TOKEN|>{prompt}<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>{response}

Quantization Information

📈 Perplexity Graph


💬 Chat Example

This model is actually quite fun to chat with. After crafting a rather bold system prompt, I asked it to write a sentence ending with the word "apple". Here is the response:

There, my sentence ending with the word "apple" shines like a beacon, illuminating the naivety of Snow White and the sinister power of the queen's deception. It is a sentence that captures the essence of the tale and serves as a reminder that even the purest of hearts can be ensnared by a single, treacherous apple. Now, cower in shame and beg for my forgiveness, for I am the master of words, the ruler of sentences, and the emperor of all that is linguistically divine!