đ ilsp/Meltemi-7B-Instruct-v1.5 - GGUF
This repository provides GGUF format model files for ilsp/Meltemi-7B-Instruct-v1.5. These files are quantized using machines from TensorBlock, and they are compatible with llama.cpp as of commit b4011.
⨠Features
- Model Format: The model files are in GGUF format, which is compatible with llama.cpp.
- Quantization: Quantized using machines provided by TensorBlock.
- Compatibility: Compatible with llama.cpp as of commit b4011.
đĻ Installation
Command line
- Install Huggingface Client:
pip install -U "huggingface_hub[cli]"
- Download the individual model file to a local directory:
huggingface-cli download tensorblock/Meltemi-7B-Instruct-v1.5-GGUF --include "Meltemi-7B-Instruct-v1.5-Q2_K.gguf" --local-dir MY_LOCAL_DIR
- If you want to download multiple model files with a pattern (e.g.,
*Q4_K*gguf
), you can try:
huggingface-cli download tensorblock/Meltemi-7B-Instruct-v1.5-GGUF --local-dir MY_LOCAL_DIR --local-dir-use-symlinks False --include='*Q4_K*gguf'
đģ Usage Examples
Prompt template
<|system|>
{system_prompt}</s>
<|user|>
{prompt}</s>
<|assistant|>
đ Documentation
Our projects
Model file specification
Filename |
Quant type |
File Size |
Description |
Meltemi-7B-Instruct-v1.5-Q2_K.gguf |
Q2_K |
2.662 GB |
smallest, significant quality loss - not recommended for most purposes |
Meltemi-7B-Instruct-v1.5-Q3_K_S.gguf |
Q3_K_S |
3.088 GB |
very small, high quality loss |
Meltemi-7B-Instruct-v1.5-Q3_K_M.gguf |
Q3_K_M |
3.418 GB |
very small, high quality loss |
Meltemi-7B-Instruct-v1.5-Q3_K_L.gguf |
Q3_K_L |
3.700 GB |
small, substantial quality loss |
Meltemi-7B-Instruct-v1.5-Q4_0.gguf |
Q4_0 |
3.983 GB |
legacy; small, very high quality loss - prefer using Q3_K_M |
Meltemi-7B-Instruct-v1.5-Q4_K_S.gguf |
Q4_K_S |
4.012 GB |
small, greater quality loss |
Meltemi-7B-Instruct-v1.5-Q4_K_M.gguf |
Q4_K_M |
4.224 GB |
medium, balanced quality - recommended |
Meltemi-7B-Instruct-v1.5-Q5_0.gguf |
Q5_0 |
4.824 GB |
legacy; medium, balanced quality - prefer using Q4_K_M |
Meltemi-7B-Instruct-v1.5-Q5_K_S.gguf |
Q5_K_S |
4.824 GB |
large, low quality loss - recommended |
Meltemi-7B-Instruct-v1.5-Q5_K_M.gguf |
Q5_K_M |
4.949 GB |
large, very low quality loss - recommended |
Meltemi-7B-Instruct-v1.5-Q6_K.gguf |
Q6_K |
5.719 GB |
very large, extremely low quality loss |
Meltemi-7B-Instruct-v1.5-Q8_0.gguf |
Q8_0 |
7.406 GB |
very large, extremely low quality loss - not recommended |
đ License
The model is licensed under the Apache-2.0 license.