Japanese Reranker Tiny V2

🚀 hotchpotch/japanese-reranker-tiny-v2

A very small and fast Japanese reranker model series (v2).

For more information about rerankers, technical reports, and evaluations, please refer to the following links:

• Release of a very small, fast, and practical Japanese reranker: japanese-reranker-tiny, xsmall v2
• Release of the highest-performing Japanese Reranker / What is a Reranker anyway?
• Technical report on creating a Japanese Reranker

🚀 Quick Start

Prerequisites

The operation requires version 4.48 or higher of the transformers library.

pip install -U "transformers>=4.48.0" sentence-transformers sentencepiece

If your GPU supports Flash Attention 2, you can install the flash-attn library for faster inference:

pip install flash-attn --no-build-isolation

💻 Usage Examples

SentenceTransformers

from sentence_transformers import CrossEncoder
import torch

MODEL_NAME = "hotchpotch/japanese-reranker-tiny-v2"

model = CrossEncoder(MODEL_NAME)
if model.device == "cuda" or model.device == "mps":
    model.model.half()
query = "感動的な映画について"
passages = [
    "深いテーマを持ちながらも、観る人の心を揺さぶる名作。登場人物の心情描写が秀逸で、ラストは涙なしでは見られない。",
    "重要なメッセージ性は評価できるが、暗い話が続くので気分が落ち込んでしまった。もう少し明るい要素があればよかった。",
    "どうにもリアリティに欠ける展開が気になった。もっと深みのある人間ドラマが見たかった。",
    "アクションシーンが楽しすぎる。見ていて飽きない。ストーリーはシンプルだが、それが逆に良い。",
]
scores = model.predict(
    [(query, passage) for passage in passages],
    show_progress_bar=True,
)
print("Scores:", scores)

SentenceTransformers + onnx

If you want to run the model faster in a CPU or ARM environment, you can use ONNX or a quantized model.

pip install onnx onnxruntime accelerate optimum

from sentence_transformers import CrossEncoder

# If you don't choose the ONNX model, model.onnx will be used automatically
# onnx_filename = None

# If you want to use the quantized optimal model, specify the file name in onnx_filename
# onnx_filename = "onnx/model_qint8_avx2.onnx"
onnx_filename = "onnx/model_qint8_arm64.onnx"

if onnx_filename:
    model = CrossEncoder(
        MODEL_NAME,
        device="cpu",
        backend="onnx",
        model_kwargs={"file_name": onnx_filename},
    )
else:
    model = CrossEncoder(MODEL_NAME, device="cpu", backend="onnx")

...

HuggingFace transformers

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from torch.nn import Sigmoid


def detect_device():
    if torch.cuda.is_available():
        return "cuda"
    elif hasattr(torch, "mps") and torch.mps.is_available():
        return "mps"
    return "cpu"


device = detect_device()
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME)
model.to(device)
model.eval()

if device == "cuda":
    model.half()

query = "感動的な映画について"
passages = [
    "深いテーマを持ちながらも、観る人の心を揺さぶる名作。登場人物の心情描写が秀逸で、ラストは涙なしでは見られない。",
    "重要なメッセージ性は評価できるが、暗い話が続くので気分が落ち込んでしまった。もう少し明るい要素があればよかった。",
    "どうにもリアリティに欠ける展開が気になった。もっと深みのある人間ドラマが見たかった。",
    "アクションシーンが楽しすぎる。見ていて飽きない。ストーリーはシンプルだが、それが逆に良い。",
]
inputs = tokenizer(
    [(query, passage) for passage in passages],
    padding=True,
    truncation=True,
    max_length=512,
    return_tensors="pt",
)
inputs = {k: v.to(device) for k, v in inputs.items()}
logits = model(**inputs).logits
activation = Sigmoid()
scores = activation(logits).squeeze().tolist()

print("Scores:", scores)

✨ Features

Characteristics of Small Rerankers

japanese-reranker-tiny-v2 and japanese-reranker-xsmall-v2 are small reranker models with the following features:

1. They can operate at a practical speed even in CPU or Apple Silicon environments.
2. They can improve the accuracy of RAG systems without expensive GPU resources.
3. They can be deployed on edge devices and used in production environments that require low latency.
4. They are based on ModernBert's ruri-v3-pt-30m.

Evaluation Results

Inference Speed

The following are the inference speed results when reranking approximately 150,000 pairs (pure model inference time excluding tokenization time). M4 Max was used for MPS (Apple Silicon) and CPU measurements, and RTX5090 was used for GPU. Flash-attention2 was used for GPU processing.

The script used for the inference speed benchmark is available here.

📄 License

This project is licensed under the MIT License.