Xlangai Jedi 3B 1080p GGUF

🚀 Llamacpp imatrix quantified version of Jedi-3B-1080p by xlangai

This project is a quantified version of xlangai's Jedi-3B-1080p model, which uses specific tools and datasets for quantification processing and can run in multiple environments, providing users with rich choices for different hardware conditions and needs.

🚀 Quick start

This project uses<a ref=“ https://github.com/ggerganov/llama.cpp/ Llama. cppRelease version<a ref= https://github.com/ggerganov/llama.cpp/releases/tag/b5524 Quantify b5524. Original model address: https://huggingface.co/xlangai/Jedi-3B-1080p All quantitative models use the imatrix option and adopt data from [here]（ https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8 ）The dataset. You can find it in LM Studio（ https://lmstudio.ai/ ）Running these quantitative models in the middle can also be done directly using [llama. cpp]（ https://github.com/ggerganov/llama.cpp ）Or any Llama.cpp based project to run.

✨ Main characteristics

-* * Multiple quantization types * : Provides a wide range of quantization types to choose from, such as bf16, Q8_0, Q6_KL, etc., to meet different performance and quality requirements. - * Specific weight processing * *: Some quantization models (such as Q3_KXL, Q4_KL, etc.) use special quantization methods to quantize the embedding and output weights into Q8_0 to improve performance. -Online repackaging: Some quantitative models support online repackaging and can automatically optimize performance based on hardware.

📦 installation guide

###Download using hugginface cli Firstly, ensure that you have installed huggingface cli:

pip install -U "huggingface_hub[cli]"

Then, you can specify the specific files to download:

huggingface-cli download bartowski/xlangai_Jedi-3B-1080p-GGUF --include "xlangai_Jedi-3B-1080p-Q4_K_M.gguf" --local-dir ./

If the model size exceeds 50GB, it will be split into multiple files. To download them all to your local folder, please run:

huggingface-cli download bartowski/xlangai_Jedi-3B-1080p-GGUF --include "xlangai_Jedi-3B-1080p-Q8_0/*" --local-dir ./

You can specify a new local directory (such as xlangai_Jedi-3B-1080p-Q8-0), or download them all to the current directory (./).

💻 Example usage

###Prompt format

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

📚 Detailed documentation

###Download file selection

🔧 Technical Details

###Quantitative tools Use<a ref=“ https://github.com/ggerganov/llama.cpp/ Llama. cppRelease version<a ref= https://github.com/ggerganov/llama.cpp/releases/tag/b5524 Quantify b5524. ###Quantitative dataset All quantitative models use the imatrix option and adopt data from [here]（ https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8 ）The dataset. ###Online repackaging Partial quantitative models support online repackaging, please refer to [this PR] for details（ https://github.com/ggerganov/llama.cpp/pull/9921 ）.

📄 permit

This project adopts Apache-2.0 license. ##Thank you Thank you to Kalomaze and Dampf for their assistance in creating the Imatrix calibration dataset. Thank you ZeroWOW for providing inspiration in embedding/outputting experiments. Thank you LM Studio for sponsoring this project. If you want to support my work, please visit my ko fi page: https://ko-fi.com/bartowski