gte-multilingual-reranker-base Open Source ONNX Version - Free Deployment, Suitable for GPU Text Classification Tasks

Gte Multilingual Reranker Base Onnx Op14 Opt Gpu Int8

Developed by JustJaro

This is the quantized ONNX version of Alibaba-NLP/gte-multilingual-reranker-base, utilizing INT8 quantization, optimized for GPU, and suitable for text classification tasks.

Text Embedding OtherOpen Source License:MIT #Multilingual Text Embedding #INT8 Quantized Inference #GPU Optimization

Downloads 91

Release Time : 3/27/2025

Model Overview

This model is a quantized ONNX version based on Alibaba-NLP/gte-multilingual-reranker-base, using ONNX opset 14, optimized for GPU devices, primarily for text classification and sentence similarity tasks.

Model Features

INT8 Quantization

Utilizes INT8 quantization technology to significantly improve inference speed.

GPU Optimization

Specially optimized for GPU devices to enhance computational efficiency.

Multilingual Support

Supports text processing tasks in multiple languages.

ONNX Runtime

Uses the ONNX runtime framework to provide efficient model inference capabilities.

Model Capabilities

Text Classification

Sentence Similarity Calculation

Multilingual Text Processing

Use Cases

Information Retrieval

Document Re-ranking

Re-rank search results in information retrieval systems to improve relevance.

Enhances the accuracy and relevance of retrieval results

Text Analysis

Text Classification

Classify text for tasks such as sentiment analysis and topic classification.

Efficient and accurate text classification

🚀 gte-multilingual-reranker-base-onnx-op14-opt-gpu-int8-quantized

This is a quantized ONNX version of Alibaba-NLP/gte-multilingual-reranker-base using ONNX opset 14, designed for text classification tasks on GPU.

🚀 Quick Start

The following code demonstrates how to load the model and perform inference:

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer

# Load model and tokenizer
model = ORTModelForSequenceClassification.from_pretrained("quantized_model")
tokenizer = AutoTokenizer.from_pretrained("quantized_model")

# Prepare input
text = "Your text here"
inputs = tokenizer(text, return_tensors="pt")

# Run inference
outputs = model(**inputs)

✨ Features

Quantization: Utilizes INT8 quantization to enhance inference speed.
Optimized for GPU: Designed to run efficiently on GPU devices.
ONNX Opset 14: Adopts the ONNX opset 14 for better compatibility and performance.

📦 Installation

The required packages and their versions are as follows:

Package	Version
transformers	4.48.3
optimum	1.24.0
onnx	1.17.0
onnxruntime	1.21.0
torch	2.5.1
numpy	1.26.4
huggingface_hub	0.28.1
python	3.12.9
system	Darwin 24.3.0

💻 Usage Examples

Basic Usage

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer

# Load model and tokenizer
model = ORTModelForSequenceClassification.from_pretrained("quantized_model")
tokenizer = AutoTokenizer.from_pretrained("quantized_model")

# Prepare input
text = "Your text here"
inputs = tokenizer(text, return_tensors="pt")

# Run inference
outputs = model(**inputs)

📚 Documentation

Model Details

Property	Details
Quantization Type	INT8
ONNX Opset	14
Task	text-classification
Target Device	GPU
Optimized	Yes
Framework	ONNX Runtime
Original Model	Alibaba-NLP/gte-multilingual-reranker-base
Quantized On	2025-03-27

Applied Optimizations

Optimization	Setting
Graph Optimization Level	Extended
Optimize for GPU	Yes
Use FP16	No
Transformers Specific Optimizations Enabled	Yes
Gelu Fusion Enabled	Yes
Layer Norm Fusion Enabled	Yes
Attention Fusion Enabled	Yes
Skip Layer Norm Fusion Enabled	Yes
Gelu Approximation Enabled	Yes

Quantization Process

This model was quantized using ONNX Runtime with int8 quantization. The quantization was performed using the Optimum library from Hugging Face with opset 14. Graph optimization was applied during export, targeting GPU devices.

Performance Comparison

Quantized models generally offer better inference speed with a slight trade-off in accuracy. This INT8 quantized model should provide significantly faster inference than the original model.

📄 License

This project is licensed under the MIT License.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご