%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
EN
Mistral NeMo Minitron 8B Base IMat GGUF
This is the result of llama.cpp imatrix quantization based on the nvidia/Mistral-NeMo-Minitron-8B-Base model, providing more options for model usage and deployment.
Downloads 1,115
Release Time : 8/21/2024
Model Overview
This model is a quantized version of NVIDIA's Mistral-NeMo-Minitron-8B-Base model, mainly used for text generation tasks.
Model Features
Multiple quantization options
Provide multiple quantization versions from 16-bit to 1-bit to meet different hardware and performance requirements.
IMatrix quantization technology
Use the imatrix quantization technology of llama.cpp to maintain better model performance at low-bit quantization levels.
GGUF format support
Adopt the GGUF format for easy deployment and use on various devices.
Model Capabilities
Text generation
Quantized model inference
Use Cases
Text generation
General text generation
Can be used to generate various types of text content
Edge device deployment
Run on low-resource devices
Run large language models on devices with limited resources through the quantized version
đ Mistral-NeMo-Minitron-8B-Base-IMat-GGUF
Llama.cpp imatrix quantization of nvidia/Mistral-NeMo-Minitron-8B-Base
This project provides a quantized version of the nvidia/Mistral-NeMo-Minitron-8B-Base model using llama.cpp's imatrix quantization. It offers various quantization types in the GGUF format for different usage scenarios.
Model Information
| Property | Details |
|---|---|
| Base Model | nvidia/Mistral-NeMo-Minitron-8B-Base |
| Original dtype | BF16 (bfloat16) |
| Quantized by | llama.cpp b3613 |
| IMatrix dataset | here |
| Library Name | gguf |
| License | nvidia-open-model-license |
| Pipeline Tag | text-generation |
| Quantized By | legraphista |
| Tags | quantized, GGUF, quantization, imat, imatrix, static, 16bit, 8bit, 6bit, 5bit, 4bit, 3bit, 2bit, 1bit |
đ Quick Start
Downloading using huggingface-cli
If you do not have hugginface-cli installed:
pip install -U "huggingface_hub[cli]"
Download the specific file you want:
huggingface-cli download legraphista/Mistral-NeMo-Minitron-8B-Base-IMat-GGUF --include "Mistral-NeMo-Minitron-8B-Base.Q8_0.gguf" --local-dir ./
If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:
huggingface-cli download legraphista/Mistral-NeMo-Minitron-8B-Base-IMat-GGUF --include "Mistral-NeMo-Minitron-8B-Base.Q8_0/*" --local-dir ./
# see FAQ for merging GGUF's
Inference
Llama.cpp
llama.cpp/main -m Mistral-NeMo-Minitron-8B-Base.Q8_0.gguf --color -i -p "prompt here"
⨠Features
- Multiple Quantization Types: Offers a wide range of quantization types including 16bit, 8bit, 6bit, 5bit, 4bit, 3bit, 2bit, and 1bit, providing flexibility for different hardware and performance requirements.
- IMatrix Quantization: Some quantizations benefit from the IMatrix input, potentially improving performance as per certain benchmarks.
đĻ Installation
The installation mainly involves downloading the model files using the huggingface-cli as described in the Quick Start section.
đģ Usage Examples
Basic Usage
Download a specific quantized model file:
huggingface-cli download legraphista/Mistral-NeMo-Minitron-8B-Base-IMat-GGUF --include "Mistral-NeMo-Minitron-8B-Base.Q8_0.gguf" --local-dir ./
Run inference using Llama.cpp:
llama.cpp/main -m Mistral-NeMo-Minitron-8B-Base.Q8_0.gguf --color -i -p "prompt here"
đ Documentation
Files
IMatrix
Status: â
Available
Link: here
Common Quants
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|---|---|---|---|---|---|
| Mistral-NeMo-Minitron-8B-Base.Q8_0.gguf | Q8_0 | 8.95GB | â Available | âĒ Static | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.Q6_K.gguf | Q6_K | 6.91GB | â Available | âĒ Static | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.Q4_K.gguf | Q4_K | 5.15GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.Q3_K.gguf | Q3_K | 4.21GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.Q2_K.gguf | Q2_K | 3.33GB | â Available | đ IMatrix | đĻ No |
All Quants
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|---|---|---|---|---|---|
| Mistral-NeMo-Minitron-8B-Base.BF16.gguf | BF16 | 16.84GB | â Available | âĒ Static | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.FP16.gguf | F16 | 16.84GB | â Available | âĒ Static | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.Q8_0.gguf | Q8_0 | 8.95GB | â Available | âĒ Static | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.Q6_K.gguf | Q6_K | 6.91GB | â Available | âĒ Static | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.Q5_K.gguf | Q5_K | 6.00GB | â Available | âĒ Static | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.Q5_K_S.gguf | Q5_K_S | 5.86GB | â Available | âĒ Static | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.Q4_K.gguf | Q4_K | 5.15GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.Q4_K_S.gguf | Q4_K_S | 4.91GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.IQ4_NL.gguf | IQ4_NL | 4.90GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.IQ4_XS.gguf | IQ4_XS | 4.66GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.Q3_K.gguf | Q3_K | 4.21GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.Q3_K_L.gguf | Q3_K_L | 4.54GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.Q3_K_S.gguf | Q3_K_S | 3.83GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.IQ3_M.gguf | IQ3_M | 3.98GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.IQ3_S.gguf | IQ3_S | 3.86GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.IQ3_XS.gguf | IQ3_XS | 3.68GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.IQ3_XXS.gguf | IQ3_XXS | 3.43GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.Q2_K.gguf | Q2_K | 3.33GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.Q2_K_S.gguf | Q2_K_S | 3.13GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.IQ2_M.gguf | IQ2_M | 3.10GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.IQ2_S.gguf | IQ2_S | 2.90GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.IQ2_XS.gguf | IQ2_XS | 2.73GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.IQ2_XXS.gguf | IQ2_XXS | 2.51GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.IQ1_M.gguf | IQ1_M | 2.27GB | â Available | đ IMatrix | đĻ No |
| Mistral-NeMo-Minitron-8B-Base.IQ1_S.gguf | IQ1_S | 2.12GB | â Available | đ IMatrix | đĻ No |
đ§ Technical Details
The quantization process is based on llama.cpp's b3613 release. The IMatrix quantization is applied selectively based on the results of this investigation, which suggests that lower quantizations benefit the most from the IMatrix input.
đ License
This project is licensed under the nvidia-open-model-license.
đ FAQ
Why is the IMatrix not applied everywhere?
According to this investigation, it appears that lower quantizations are the only ones that benefit from the imatrix input (as per hellaswag results).
How do I merge a split GGUF?
- Make sure you have
gguf-splitavailable- To get hold of
gguf-split, navigate to https://github.com/ggerganov/llama.cpp/releases - Download the appropriate zip for your system from the latest release
- Unzip the archive and you should be able to find
gguf-split
- To get hold of
- Locate your GGUF chunks folder (ex:
Mistral-NeMo-Minitron-8B-Base.Q8_0) - Run
gguf-split --merge Mistral-NeMo-Minitron-8B-Base.Q8_0/Mistral-NeMo-Minitron-8B-Base.Q8_0-00001-of-XXXXX.gguf Mistral-NeMo-Minitron-8B-Base.Q8_0.gguf- Make sure to point
gguf-splitto the first chunk of the split.
- Make sure to point
Got a suggestion? Ping me @legraphista!
Phi 2 GGUF
Other
Phi-2 is a small yet powerful language model developed by Microsoft, featuring 2.7 billion parameters, focusing on efficient inference and high-quality text generation.
Large Language Model Supports Multiple Languages
P
TheBloke
41.5M
205
Roberta Large
MIT
A large English language model pre-trained with masked language modeling objectives, using improved BERT training methods
Large Language Model English
R
FacebookAI
19.4M
212
Distilbert Base Uncased
Apache-2.0
DistilBERT is a distilled version of the BERT base model, maintaining similar performance while being more lightweight and efficient, suitable for natural language processing tasks such as sequence classification and token classification.
Large Language Model English
D
distilbert
11.1M
669
Llama 3.1 8B Instruct GGUF
Meta Llama 3.1 8B Instruct is a multilingual large language model optimized for multilingual dialogue use cases, excelling in common industry benchmarks.
Large Language Model English
L
modularai
9.7M
4
Xlm Roberta Base
MIT
XLM-RoBERTa is a multilingual model pretrained on 2.5TB of filtered CommonCrawl data across 100 languages, using masked language modeling as the training objective.
Large Language Model Supports Multiple Languages
X
FacebookAI
9.6M
664
Roberta Base
MIT
An English pre-trained model based on Transformer architecture, trained on massive text through masked language modeling objectives, supporting text feature extraction and downstream task fine-tuning
Large Language Model English
R
FacebookAI
9.3M
488
Opt 125m
Other
OPT is an open pre-trained Transformer language model suite released by Meta AI, with parameter sizes ranging from 125 million to 175 billion, designed to match the performance of the GPT-3 series while promoting open research in large-scale language models.
Large Language Model English
O
facebook
6.3M
198
1
A pretrained model based on the transformers library, suitable for various NLP tasks
Large Language Model
Transformers
Transformers1
unslothai
6.2M
1
Llama 3.1 8B Instruct
Llama 3.1 is Meta's multilingual large language model series, featuring 8B, 70B, and 405B parameter scales, supporting 8 languages and code generation, with optimized multilingual dialogue scenarios.
Large Language Model Transformers Supports Multiple Languages
Transformers Supports Multiple LanguagesL
meta-llama
5.7M
3,898
T5 Base
Apache-2.0
The T5 Base Version is a text-to-text Transformer model developed by Google with 220 million parameters, supporting multilingual NLP tasks.
Large Language Model Supports Multiple Languages
T
google-t5
5.4M
702
Featured Recommended AI Models
Š 2025AIbase