rank_zephyr_7b_v1_full-GGUF Open-Source Model - Free for Efficient Text Ranking Tasks

Rank Zephyr 7b V1 Full GGUF

Developed by tensorblock

This is the GGUF quantized version of the castorini/rank_zephyr_7b_v1_full model, designed for text ranking tasks.

Large Language Model EnglishOpen Source License:MIT #Text Ranking #Lightweight Inference #Multi-level Quantization

Downloads 66

Release Time : 11/16/2024

Model Overview

This model is a quantized version of Zephyr 7B, specifically tailored for text ranking tasks, offering multiple quantization options to suit various hardware requirements.

Model Features

Multiple Quantization Options

Offers 12 quantization levels from Q2_K to Q8_0 to meet usage needs under different hardware conditions

Efficient Inference

Optimized GGUF format enhances inference efficiency while maintaining model performance

Strong Compatibility

Compatible with llama.cpp and can run on various devices

Model Capabilities

Text Ranking

Text Understanding

Contextual Reasoning

Use Cases

Information Retrieval

Search Result Ranking

Rank the relevance of search engine results

Improves search result relevance and user experience

Recommendation Systems

castorini/rank_zephyr_7b_v1_full - GGUF

This repo contains GGUF format model files for castorini/rank_zephyr_7b_v1_full.

The files were quantized using machines provided by TensorBlock, and they are compatible with llama.cpp as of commit b4011.

Our projects

Awesome MCP Servers	TensorBlock Studio

A comprehensive collection of Model Context Protocol (MCP) servers.	A lightweight, open, and extensible multi-LLM interaction studio.
👀 See what we built 👀	👀 See what we built 👀

## Prompt template

<|system|>
{system_prompt}</s>
<|user|>
{prompt}</s>
<|assistant|>

Model file specification

Filename	Quant type	File Size	Description
rank_zephyr_7b_v1_full-Q2_K.gguf	Q2_K	2.532 GB	smallest, significant quality loss - not recommended for most purposes
rank_zephyr_7b_v1_full-Q3_K_S.gguf	Q3_K_S	2.947 GB	very small, high quality loss
rank_zephyr_7b_v1_full-Q3_K_M.gguf	Q3_K_M	3.277 GB	very small, high quality loss
rank_zephyr_7b_v1_full-Q3_K_L.gguf	Q3_K_L	3.560 GB	small, substantial quality loss
rank_zephyr_7b_v1_full-Q4_0.gguf	Q4_0	3.827 GB	legacy; small, very high quality loss - prefer using Q3_K_M
rank_zephyr_7b_v1_full-Q4_K_S.gguf	Q4_K_S	3.856 GB	small, greater quality loss
rank_zephyr_7b_v1_full-Q4_K_M.gguf	Q4_K_M	4.068 GB	medium, balanced quality - recommended
rank_zephyr_7b_v1_full-Q5_0.gguf	Q5_0	4.654 GB	legacy; medium, balanced quality - prefer using Q4_K_M
rank_zephyr_7b_v1_full-Q5_K_S.gguf	Q5_K_S	4.654 GB	large, low quality loss - recommended
rank_zephyr_7b_v1_full-Q5_K_M.gguf	Q5_K_M	4.779 GB	large, very low quality loss - recommended
rank_zephyr_7b_v1_full-Q6_K.gguf	Q6_K	5.534 GB	very large, extremely low quality loss
rank_zephyr_7b_v1_full-Q8_0.gguf	Q8_0	7.167 GB	very large, extremely low quality loss - not recommended

Downloading instruction

Command line

Firstly, install Huggingface Client

pip install -U "huggingface_hub[cli]"

Then, downoad the individual model file the a local directory

huggingface-cli download tensorblock/rank_zephyr_7b_v1_full-GGUF --include "rank_zephyr_7b_v1_full-Q2_K.gguf" --local-dir MY_LOCAL_DIR

If you wanna download multiple model files with a pattern (e.g., *Q4_K*gguf), you can try:

huggingface-cli download tensorblock/rank_zephyr_7b_v1_full-GGUF --local-dir MY_LOCAL_DIR --local-dir-use-symlinks False --include='*Q4_K*gguf'

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご