Model Overview
Model Features
Model Capabilities
Use Cases
đ Claire 7B 0.1 - GGUF
This repository offers GGUF format model files for OpenLLM France's Claire 7B 0.1, facilitating various text - generation tasks.
đĻ Model Information
Property | Details |
---|---|
Base Model | OpenLLM-France/Claire-7B-0.1 |
Inference | false |
Language | fr |
License | cc-by-nc-sa-4.0 |
Model Creator | OpenLLM France |
Model Name | Claire 7B 0.1 |
Model Type | falcon |
Pipeline Tag | text-generation |
Quantized By | TheBloke |
Tags | pretrained, conversational |
đ Visual Introduction

TheBloke's LLM work is generously supported by a grant from andreessen horowitz (a16z)
đ Description
This repo contains GGUF format model files for OpenLLM France's Claire 7B 0.1.
These files were quantised using hardware kindly provided by Massed Compute.
đĄ About GGUF
GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.
Here is an incomplete list of clients and libraries that are known to support GGUF:
- llama.cpp. The source project for GGUF. Offers a CLI and a server option.
- text-generation-webui, the most widely used web UI, with many features and powerful extensions. Supports GPU acceleration.
- KoboldCpp, a fully featured web UI, with GPU accel across all platforms and GPU architectures. Especially good for story telling.
- LM Studio, an easy - to - use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration.
- LoLLMS Web UI, a great web UI with many interesting and unique features, including a full model library for easy model selection.
- Faraday.dev, an attractive and easy to use character - based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
- ctransformers, a Python library with GPU accel, LangChain support, and OpenAI - compatible AI server.
- llama - cpp - python, a Python library with GPU accel, LangChain support, and OpenAI - compatible API server.
- candle, a Rust ML framework with a focus on performance, including GPU support, and ease of use.
đ Repositories available
- AWQ model(s) for GPU inference.
- GPTQ models for GPU inference, with multiple quantisation parameter options.
- 2, 3, 4, 5, 6 and 8 - bit GGUF models for CPU+GPU inference
- OpenLLM France's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions
đ Prompt template: OpenLLM - France
- Bonjour BotName, {prompt}
- Bonjour UserName,
đ ī¸ Compatibility
These quantised GGUFv2 files are compatible with llama.cpp from August 27th onwards, as of commit d0cee0d
They are also compatible with many third party UIs and libraries - please see the list at the top of this README.
đ Explanation of quantisation methods
Click to see details
The new methods available are:
- GGML_TYPE_Q2_K - "type - 1" 2 - bit quantization in super - blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
- GGML_TYPE_Q3_K - "type - 0" 3 - bit quantization in super - blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
- GGML_TYPE_Q4_K - "type - 1" 4 - bit quantization in super - blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
- GGML_TYPE_Q5_K - "type - 1" 5 - bit quantization. Same super - block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
- GGML_TYPE_Q6_K - "type - 0" 6 - bit quantization. Super - blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
Refer to the Provided Files table below to see what files use which methods, and how.
đ Provided files
Name | Quant method | Bits | Size | Max RAM required | Use case |
---|---|---|---|---|---|
claire-7b-0.1.Q2_K.gguf | Q2_K | 2 | 4.02 GB | 6.52 GB | smallest, significant quality loss - not recommended for most purposes |
claire-7b-0.1.Q3_K_S.gguf | Q3_K_S | 3 | 4.13 GB | 6.63 GB | very small, high quality loss |
claire-7b-0.1.Q4_0.gguf | Q4_0 | 4 | 4.21 GB | 6.71 GB | legacy; small, very high quality loss - prefer using Q3_K_M |
claire-7b-0.1.Q3_K_M.gguf | Q3_K_M | 3 | 4.37 GB | 6.87 GB | very small, high quality loss |
claire-7b-0.1.Q3_K_L.gguf | Q3_K_L | 3 | 4.56 GB | 7.06 GB | small, substantial quality loss |
claire-7b-0.1.Q4_K_S.gguf | Q4_K_S | 4 | 4.75 GB | 7.25 GB | small, greater quality loss |
claire-7b-0.1.Q4_K_M.gguf | Q4_K_M | 4 | 4.98 GB | 7.48 GB | medium, balanced quality - recommended |
claire-7b-0.1.Q5_0.gguf | Q5_0 | 5 | 5.08 GB | 7.58 GB | legacy; medium, balanced quality - prefer using Q4_K_M |
claire-7b-0.1.Q5_K_S.gguf | Q5_K_S | 5 | 5.34 GB | 7.84 GB | large, low quality loss - recommended |
claire-7b-0.1.Q5_K_M.gguf | Q5_K_M | 5 | 5.73 GB | 8.23 GB | large, very low quality loss - recommended |
claire-7b-0.1.Q6_K.gguf | Q6_K | 6 | 7.03 GB | 9.53 GB | very large, extremely low quality loss |
claire-7b-0.1.Q8_0.gguf | Q8_0 | 8 | 7.67 GB | 10.17 GB | very large, extremely low quality loss - not recommended |
Note: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
đĨ How to download GGUF files
â ī¸ Important Note
You almost never want to clone the entire repo! Multiple different quantisation formats are provided, and most users only want to pick and download a single file.
The following clients/libraries will automatically download models for you, providing a list of available models to choose from:
- LM Studio
- LoLLMS Web UI
- Faraday.dev
In text - generation - webui
Under Download Model, you can enter the model repo: TheBloke/Claire-7B-0.1-GGUF and below it, a specific filename to download, such as: claire-7b-0.1.Q4_K_M.gguf.
Then click Download.
On the command line, including multiple files at once
I recommend using the huggingface - hub
Python library:
pip3 install huggingface-hub
Then you can download any individual model file to the current directory, at high speed, with a command like this:
huggingface-cli download TheBloke/Claire-7B-0.1-GGUF claire-7b-0.1.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
More advanced huggingface - cli download usage
You can also download multiple files at once with a pattern:
huggingface-cli download TheBloke/Claire-7B-0.1-GGUF --local-dir . --local-dir-use-symlinks False --include='*Q4_K*gguf'
For more documentation on downloading with huggingface - cli
, please see: HF -> Hub Python Library -> Download files -> Download from the CLI.
To accelerate downloads on fast connections (1Gbit/s or higher), install hf_transfer
:
pip3 install hf_transfer
And set environment variable HF_HUB_ENABLE_HF_TRANSFER
to 1
:
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download TheBloke/Claire-7B-0.1-GGUF claire-7b-0.1.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
Windows Command Line users: You can set the environment variable by running set HF_HUB_ENABLE_HF_TRANSFER=1
before the download command.
đ Example llama.cpp
command
Make sure you are using llama.cpp
from commit d0cee0d or later.
./main -ngl 32 -m claire-7b-0.1.Q4_K_M.gguf --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "- Bonjour BotName, {prompt}\n- Bonjour UserName,"
Change -ngl 32
to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
Change -c 2048
to the desired sequence length. For extended sequence models - eg 8K, 16K, 32K - the necessary RoPE scaling parameters are read from the GGUF file and set by llama.cpp automatically.
If you want to have a chat - style conversation, replace the -p <PROMPT>
argument with -i -ins
For other parameters and how to use them, please refer to the llama.cpp documentation
đĨī¸ How to run in text - generation - webui
Further instructions can be found in the text - generation - webui documentation, here: text - generation - webui/docs/04 â Model Tab.md.
đ How to run from Python code
You can use GGUF models from Python using the llama - cpp - python or ctransformers libraries.

