Midm 2.0 Base Instruct GGUF

🚀 Midm-2.0-Base-Instruct GGUF Models

This project offers the Midm-2.0-Base-Instruct GGUF models, which are designed for text generation tasks. These models are developed with a focus on specific quantization techniques and provide high - performance in relevant evaluations.

🚀 Quick Start

Here is the code snippet to run conversational inference with the model:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig

model_name = "K-intelligence/Midm-2.0-Base-Instruct"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
generation_config = GenerationConfig.from_pretrained(model_name)

prompt = "KT에 대해 소개해줘"

# message for inference
messages = [
    {"role": "system", 
     "content": "Mi:dm(믿:음)은 KT에서 개발한 AI 기반 어시스턴트이다."},
    {"role": "user", "content": prompt}
]

input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
)

output = model.generate(
    input_ids.to("cuda"),
    generation_config=generation_config,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=128,
    do_sample=False,
)
print(tokenizer.decode(output[0]))

⚠️ Important Note

The transformers library should be version 4.45.0 or higher.

✨ Features

• Korea - centric AI: Mi:dm 2.0 is a "Korea - centric AI" model that deeply internalizes the unique values, cognitive frameworks, and commonsense reasoning of Korean society.
• Two Versions: It is released in two versions, the 11.5B parameter dense Mi:dm 2.0 Base for balanced performance and the 2.3B parameter dense Mi:dm 2.0 Mini for on - device and limited - GPU environments.
• New Quantization Approach: A new quantization approach is used to selectively elevate the precision of key layers, improving precision for a given quantization level.

📦 Installation

To serve Mi:dm 2.0 using vLLM(>=0.8.0) with an OpenAI - compatible API:

vllm serve K - intelligence/Midm - 2.0 - Base - Instruct

💻 Usage Examples

Run on Friendli.AI

You can try our model immediately via Friendli.AI. Simply click Deploy and then Friendli Endpoints.

⚠️ Important Note

Please note that a login to Friendli.AI is required after your fifth chat interaction.

Run on Your Local Machine

We provide a detailed description about running Mi:dm 2.0 on your local machine using llama.cpp, LM Studio, and Ollama. Please check our github for more information

Deployment

vllm serve K - intelligence/Midm - 2.0 - Base - Instruct

Tutorials

To help our end - users easily use Mi:dm 2.0, we have provided comprehensive tutorials on github.

📚 Documentation

Model Generation Details

This model was generated using llama.cpp at commit 21c02174.

Quantization Beyond the IMatrix

I've been experimenting with a new quantization approach that selectively elevates the precision of key layers beyond what the default IMatrix configuration provides.

In my testing, standard IMatrix quantization underperforms at lower bit depths, especially with Mixture of Experts (MoE) models. To address this, I'm using the --tensor - type option in llama.cpp to manually "bump" important layers to higher precision. You can see the implementation here:
👉 [Layer bumping with llama.cpp](https://github.com/Mungert69/GGUFModelBuilder/blob/main/model - converter/tensor_list_builder.py)

While this does increase model file size, it significantly improves precision for a given quantization level.

Evaluation

Korean

English

🔧 Technical Details

The model is developed with a focus on specific quantization techniques and is based on the llama.cpp framework. The new quantization approach helps to improve the precision of the model, especially for Mixture of Experts (MoE) models.

📄 License

Mi:dm 2.0 is licensed under the MIT License.

📚 If you find these models useful

Help me test my AI - Powered Quantum Network Monitor Assistant with quantum - ready security checks:

👉 Quantum Network Monitor

The full Open Source Code for the Quantum Network Monitor Service available at my github repos (repos with NetworkMonitor in the name): Source Code Quantum Network Monitor. You will also find the code I use to quantize the models if you want to do it yourself GGUFModelBuilder

💬 How to test:
Choose an AI assistant type:

• TurboLLM (GPT - 4.1 - mini)
• HugLLM (Hugginface Open - source models)
• TestLLM (Experimental CPU - only)

What I’m Testing

I’m pushing the limits of small open - source models for AI network monitoring, specifically:

• Function calling against live network services
• How small can a model go while still handling:
  • Automated Nmap security scans
  • Quantum - readiness checks
  • Network Monitoring tasks

🟡 TestLLM – Current experimental model (llama.cpp on 2 CPU threads on huggingface docker space):

• ✅ Zero - configuration setup
• ⏳ 30s load time (slow inference but no API costs). No token limited as the cost is low.
• 🔧 Help wanted! If you’re into edge - device AI, let’s collaborate