đ Medra27B Quantized Model
This project provides quantized versions of the Medra27B model, offering various quantization options for different use - cases in medical AI and text generation.
đ Quick Start
If you are new to using GGUF files, refer to TheBloke's READMEs for comprehensive details, including instructions on how to concatenate multi - part files.
⨠Features
- Multiple Quantization Options: Offers a wide range of quantized versions sorted by size, suitable for different performance and quality requirements.
- Medical - AI Focus: Based on a medical - related base model, useful for medical text generation, summarization, and diagnostic reasoning.
- Fine - Tuned: The model is fine - tuned, enhancing its performance in specific tasks.
đĻ Installation
No specific installation steps are provided in the original document.
đģ Usage Examples
No code examples are provided in the original document.
đ Documentation
About
The model is a weighted/imatrix quantization of https://huggingface.co/nicoboss/Medra27B. Static quants are available at https://huggingface.co/mradermacher/Medra27B - GGUF.
Provided Quants
The provided quantized models are sorted by size (not necessarily quality). IQ - quants are often preferable over similar - sized non - IQ quants.
| Link |
Type |
Size/GB |
Notes |
| GGUF |
i1 - IQ1_S |
6.4 |
for the desperate |
| GGUF |
i1 - IQ1_M |
6.9 |
mostly desperate |
| GGUF |
i1 - IQ2_XXS |
7.8 |
|
| GGUF |
i1 - IQ2_XS |
8.5 |
|
| GGUF |
i1 - IQ2_S |
8.9 |
|
| GGUF |
i1 - IQ2_M |
9.6 |
|
| GGUF |
i1 - Q2_K_S |
9.9 |
very low quality |
| GGUF |
i1 - Q2_K |
10.6 |
IQ3_XXS probably better |
| GGUF |
i1 - IQ3_XXS |
10.8 |
lower quality |
| GGUF |
i1 - IQ3_XS |
11.7 |
|
| GGUF |
i1 - IQ3_S |
12.3 |
beats Q3_K* |
| GGUF |
i1 - Q3_K_S |
12.3 |
IQ3_XS probably better |
| GGUF |
i1 - IQ3_M |
12.6 |
|
| GGUF |
i1 - Q3_K_M |
13.5 |
IQ3_S probably better |
| GGUF |
i1 - Q3_K_L |
14.6 |
IQ3_M probably better |
| GGUF |
i1 - IQ4_XS |
14.9 |
|
| GGUF |
i1 - Q4_0 |
15.7 |
fast, low quality |
| GGUF |
i1 - Q4_K_S |
15.8 |
optimal size/speed/quality |
| GGUF |
i1 - Q4_K_M |
16.6 |
fast, recommended |
| GGUF |
i1 - Q4_1 |
17.3 |
|
| GGUF |
i1 - Q5_K_S |
18.9 |
|
| GGUF |
i1 - Q5_K_M |
19.4 |
|
| GGUF |
i1 - Q6_K |
22.3 |
practically like static Q6_K |
Here is a useful graph by ikawrakow comparing some lower - quality quant types (lower is better):
And here are Artefact2's thoughts on the matter:
https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9
FAQ / Model Request
For answers to common questions or if you want other models to be quantized, visit https://huggingface.co/mradermacher/model_requests.
đ§ Technical Details
No technical details are provided in the original document.
đ License
The model is licensed under the apache - 2.0 license.
Thanks
I express my gratitude to my company, nethype GmbH, for allowing me to use its servers and upgrading my workstation, which enables me to carry out this work in my free time. I also thank @nicoboss for granting me access to his private supercomputer, which allows me to provide many more imatrix quants with much higher quality than I could otherwise.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
