IBM Granite 8B Open-source Code Instruction Model - Supports long contexts and enables free code generation and instruction understanding

Granite 8b Code Instruct 128k GGUF

Developed by tensorblock

IBM Granite 8B code instruction model, supporting a context length of 128k, focusing on code generation and instruction understanding tasks.

Large Language Model

Transformers

OtherOpen Source License:Apache-2.0 #Code generation #Long context support #Multi-task instructions

Downloads 186

Release Time : 11/14/2024

Model Overview

This is a large language model based on the Transformer architecture, specifically optimized for code generation and instruction understanding tasks. The model supports multiple programming languages and mathematical reasoning tasks.

Model Features

Long context support

Supports a context length of 128k, suitable for handling long code files and complex instructions

Multi-task optimization

Specifically optimized for multiple tasks such as code generation, mathematical reasoning, and instruction understanding

Multiple quantization options

Provides multiple quantization versions from Q2_K to Q8_0 to meet deployment requirements under different hardware conditions

Compatible with llama.cpp

The GGUF format model file is compatible with llama.cpp, facilitating deployment and use

Model Capabilities

Code generation

Instruction understanding

Mathematical reasoning

SQL generation

Function call

Use Cases

Programming assistance

Python code completion

Generate Python code snippets based on natural language descriptions

HumanEvalSynthesis pass@1 62.2

SQL query generation

Generate SQL query statements based on natural language descriptions

Mathematical problem solving

Mathematical reasoning

Solve complex mathematical problems and proofs

🚀 ibm-granite/granite-8b-code-instruct-128k - GGUF

This repository stores GGUF format model files for ibm-granite/granite-8b-code-instruct-128k. The files were quantized using machines provided by TensorBlock, and they are compatible with llama.cpp as of commit b4011.

✨ Features

Diverse Datasets: Trained on multiple datasets such as bigcode/commitpackft, TIGER-Lab/MathInstruct, etc.
Multiple Metrics: Evaluated using metrics like code_eval.
Compatible Format: GGUF format is compatible with llama.cpp.

📦 Installation

Command line

Firstly, install Huggingface Client:

pip install -U "huggingface_hub[cli]"

Then, download the individual model file to a local directory:

huggingface-cli download tensorblock/granite-8b-code-instruct-128k-GGUF --include "granite-8b-code-instruct-128k-Q2_K.gguf" --local-dir MY_LOCAL_DIR

If you want to download multiple model files with a pattern (e.g., *Q4_K*gguf), you can try:

huggingface-cli download tensorblock/granite-8b-code-instruct-128k-GGUF --local-dir MY_LOCAL_DIR --local-dir-use-symlinks False --include='*Q4_K*gguf'

📚 Documentation

Our projects

Project	Description	Image	Link
Awesome MCP Servers	A comprehensive collection of Model Context Protocol (MCP) servers.		See what we built
TensorBlock Studio	A lightweight, open, and extensible multi-LLM interaction studio.		See what we built

Prompt template

System:
{system_prompt}

Question:
{prompt}

Answer:

Model file specification

Filename	Quant type	File Size	Description
granite-8b-code-instruct-128k-Q2_K.gguf	Q2_K	2.852 GB	smallest, significant quality loss - not recommended for most purposes
granite-8b-code-instruct-128k-Q3_K_S.gguf	Q3_K_S	3.304 GB	very small, high quality loss
granite-8b-code-instruct-128k-Q3_K_M.gguf	Q3_K_M	3.674 GB	very small, high quality loss
granite-8b-code-instruct-128k-Q3_K_L.gguf	Q3_K_L	3.993 GB	small, substantial quality loss
granite-8b-code-instruct-128k-Q4_0.gguf	Q4_0	4.276 GB	legacy; small, very high quality loss - prefer using Q3_K_M
granite-8b-code-instruct-128k-Q4_K_S.gguf	Q4_K_S	4.305 GB	small, greater quality loss
granite-8b-code-instruct-128k-Q4_K_M.gguf	Q4_K_M	4.548 GB	medium, balanced quality - recommended
granite-8b-code-instruct-128k-Q5_0.gguf	Q5_0	5.190 GB	legacy; medium, balanced quality - prefer using Q4_K_M
granite-8b-code-instruct-128k-Q5_K_S.gguf	Q5_K_S	5.190 GB	large, low quality loss - recommended
granite-8b-code-instruct-128k-Q5_K_M.gguf	Q5_K_M	5.330 GB	large, very low quality loss - recommended
granite-8b-code-instruct-128k-Q6_K.gguf	Q6_K	6.161 GB	very large, extremely low quality loss
granite-8b-code-instruct-128k-Q8_0.gguf	Q8_0	7.977 GB	very large, extremely low quality loss - not recommended

📄 License

This project is licensed under the apache-2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご