gte - Qwen2 - 7B - instruct - GGUF Open - source Multilingual Text Embedding Model

Gte Qwen2 7B Instruct GGUF

Developed by mradermacher

A 7B-parameter multilingual text embedding model developed by Alibaba NLP team, specializing in sentence similarity tasks, offering multiple quantization versions

Large Language Model EnglishOpen Source License:Apache-2.0 #English sentence embedding #High-precision semantic matching #Lightweight deployment

Downloads 510

Release Time : 2/16/2025

Model Overview

A 7B-parameter model based on Qwen2 architecture, primarily used for sentence similarity calculation and text embedding tasks, supporting English processing

Model Features

Multiple quantization versions

Provides 12 different precision quantization versions from Q2_K to f16 to meet various scenario requirements

Efficient inference

After quantization, the model size is significantly reduced, with the minimum being only 3.1GB (Q2_K), suitable for resource-constrained environments

High-quality embeddings

Performs excellently in benchmarks like MTEB, capable of generating high-quality sentence embeddings

Model Capabilities

Sentence embedding generation

Text similarity calculation

Semantic search

Use Cases

Information retrieval

Document similarity search

Quickly find semantically similar documents in large-scale document libraries

Recommendation systems

Content recommendation

Generate personalized recommendations based on content semantic similarity

🚀 gte-Qwen2-7B-instruct-GGUF

This repository provides static quantizations of the gte-Qwen2-7B-instruct model, offering various quantization options for different use cases.

🚀 Quick Start

If you are unsure how to use GGUF files, refer to one of TheBloke's READMEs for more details, including on how to concatenate multi-part files.

✨ Features

Quantization: Static quants of https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct are provided.
Multiple Quant Types: A variety of quantization types are available, sorted by size.

📦 Information

Property	Details
Base Model	Alibaba-NLP/gte-Qwen2-7B-instruct
Language	en
Library Name	transformers
License	apache-2.0
Quantized By	mradermacher
Tags	mteb, sentence-transformers, transformers, Qwen2, sentence-similarity

📚 Documentation

About

Static quants of https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct. Weighted/imatrix quants seem not to be available (by me) at this time. If they do not show up a week or so after the static ones, I have probably not planned for them. Feel free to request them by opening a Community Discussion.

Provided Quants

(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants)

Link	Type	Size/GB	Notes
GGUF	Q2_K	3.1
GGUF	Q3_K_S	3.6
GGUF	Q3_K_M	3.9	lower quality
GGUF	Q3_K_L	4.2
GGUF	IQ4_XS	4.3
GGUF	Q4_K_S	4.6	fast, recommended
GGUF	Q4_K_M	4.8	fast, recommended
GGUF	Q5_K_S	5.4
GGUF	Q5_K_M	5.5
GGUF	Q6_K	6.4	very good quality
GGUF	Q8_0	8.2	fast, best quality
GGUF	f16	15.3	16 bpw, overkill

Here is a handy graph by ikawrakow comparing some lower-quality quant types (lower is better):

And here are Artefact2's thoughts on the matter: https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9

FAQ / Model Request

See https://huggingface.co/mradermacher/model_requests for some answers to questions you might have and/or if you want some other model quantized.

📄 License

This project is licensed under the apache-2.0 license.

🙏 Thanks

I thank my company, nethype GmbH, for letting me use its servers and providing upgrades to my workstation to enable this work in my free time. Additional thanks to @nicoboss for giving me access to his private supercomputer, enabling me to provide many more imatrix quants, at much higher quality, than I would otherwise be able to.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご