Glucose Base Ja V2

🚀 GLuCoSE v2

GLuCoSE v2 is a general Japanese text embedding model designed for retrieval tasks. It can run on CPU and effectively measure semantic similarity between sentences.

🚀 Quick Start

GLuCoSE v2 is a powerful Japanese text embedding model. It can be used to measure semantic similarity between sentences and serve as a retrieval system for searching passages based on queries. During inference, each input text should start with "query: " or "passage: ".

✨ Features

• Specialized for Retrieval: Demonstrates the highest performance among similar size models in MIRACL and other tasks.
• Japanese Text Optimization: Optimized for Japanese text processing.
• CPU Compatibility: Can run on CPU.

📚 Documentation

Model Description

The model is based on GLuCoSE and fine - tuned through distillation using several large - scale embedding models and multi - stage contrastive learning.

• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity

Usage

Direct Usage (Sentence Transformers)

You can perform inference using SentenceTransformer with the following code:

from sentence_transformers import SentenceTransformer
import torch.nn.functional as F

# Download from the 🤗 Hub
model = SentenceTransformer("pkshatech/GLuCoSE-base-ja-v2")

# Each input text should start with "query: " or "passage: ".
# For tasks other than retrieval, you can simply use the "query: " prefix.
sentences = [
    'query: PKSHAはどんな会社ですか？',
    'passage: 研究開発したアルゴリズムを、多くの企業のソフトウエア・オペレーションに導入しています。',
    'query: 日本で一番高い山は？',
    'passage: 富士山（ふじさん）は、標高3776.12 m、日本最高峰（剣ヶ峰）の独立峰で、その優美な風貌は日本国外でも日本の象徴として広く知られている。',
]
embeddings = model.encode(sentences,convert_to_tensor=True)
print(embeddings.shape)
# [4, 768]

# Get the similarity scores for the embeddings
similarities = F.cosine_similarity(embeddings.unsqueeze(0), embeddings.unsqueeze(1), dim=2)
print(similarities)
# [[1.0000, 0.6050, 0.4341, 0.5537],
# [0.6050, 1.0000, 0.5018, 0.6815],
# [0.4341, 0.5018, 1.0000, 0.7534],
# [0.5537, 0.6815, 0.7534, 1.0000]]

Direct Usage (Transformers)

You can perform inference using Transformers with the following code:

import torch.nn.functional as F
from torch import Tensor
from transformers import AutoTokenizer, AutoModel

def mean_pooling(last_hidden_states: Tensor,attention_mask: Tensor) -> Tensor:
    emb = last_hidden_states * attention_mask.unsqueeze(-1)
    emb = emb.sum(dim=1) / attention_mask.sum(dim=1).unsqueeze(-1)
    return emb

# Download from the 🤗 Hub
tokenizer = AutoTokenizer.from_pretrained("pkshatech/GLuCoSE-base-ja-v2")
model = AutoModel.from_pretrained("pkshatech/GLuCoSE-base-ja-v2")

# Each input text should start with "query: " or "passage: ".
# For tasks other than retrieval, you can simply use the "query: " prefix.
sentences = [
    'query: PKSHAはどんな会社ですか？',
    'passage: 研究開発したアルゴリズムを、多くの企業のソフトウエア・オペレーションに導入しています。',
    'query: 日本で一番高い山は？',
    'passage: 富士山（ふじさん）は、標高3776.12 m、日本最高峰（剣ヶ峰）の独立峰で、その優美な風貌は日本国外でも日本の象徴として広く知られている。',
]

# Tokenize the input texts
batch_dict = tokenizer(sentences, max_length=512, padding=True, truncation=True, return_tensors='pt')

outputs = model(**batch_dict)
embeddings = mean_pooling(outputs.last_hidden_state, batch_dict['attention_mask'])
print(embeddings.shape)
# [4, 768]

# Get the similarity scores for the embeddings
similarities = F.cosine_similarity(embeddings.unsqueeze(0), embeddings.unsqueeze(1), dim=2)
print(similarities)
# [[1.0000, 0.6050, 0.4341, 0.5537],
# [0.6050, 1.0000, 0.5018, 0.6815],
# [0.4341, 0.5018, 1.0000, 0.7534],
# [0.5537, 0.6815, 0.7534, 1.0000]]

Training Details

The fine - tuning of GLuCoSE v2 is carried out through the following steps: Step 1: Ensemble distillation

• The embedded representation was distilled using [E5 - mistral](https://huggingface.co/intfloat/e5 - mistral - 7b - instruct), [gte - Qwen2](https://huggingface.co/Alibaba - NLP/gte - Qwen2 - 7B - instruct), and [mE5 - large](https://huggingface.co/intfloat/multilingual - e5 - large) as teacher models.

Step 2: Contrastive learning

• Triplets were created from [JSNLI](https://nlp.ist.i.kyoto - u.ac.jp/?%E6%97%A5%E6%9C%AC%E8%AA%9ESNLI%28JSNLI%29%E3%83%87%E3%83%BC%E3%82%BF%E3%82%BB%E3%83%83%E3%83%88), [MNLI](https://huggingface.co/datasets/MoritzLaurer/multilingual - NLI - 26lang - 2mil7), [PAWS - X](https://huggingface.co/datasets/paws - x), JSeM and [Mr.TyDi](https://huggingface.co/datasets/castorini/mr - tydi) and used for training.
• This training aimed to improve the overall performance as a sentence embedding model.

Step 3: Search - specific contrastive learning

• In order to make the model more robust to the retrieval task, additional two - stage training with QA and retrieval task was conducted.
• In the first stage, the synthetic dataset [auto - wiki - qa](https://huggingface.co/datasets/cl - nagoya/auto - wiki - qa) was used for training, while in the second stage, JQaRA, [MQA](https://huggingface.co/datasets/hpprc/mqa - ja), Japanese Wikipedia Human Retrieval, Mr.TyDi,MIRACL, Quiz Works and Quiz No Mori were used.

Benchmarks

Retrieval

Evaluated with MIRACL - ja, JQARA, JaCWIR and MLDR - ja.

Note: Results for OpenAI small embeddings in JQARA and JaCWIR are quoted from the JQARA and JaCWIR.

JMTEB

Evaluated with JMTEB. The average score is macro - average.

Note: Results for OpenAI embeddings and multilingual - e5 models are quoted from the JMTEB leaderboard. Results for ruri are quoted from the [cl - nagoya/ruri - base model card](https://huggingface.co/cl - nagoya/ruri - base/blob/main/README.md).

👥 Authors

Chihiro Yano, Mocho Go, Hideyuki Tachibana, Hiroto Takegawa, Yotaro Watanabe

📄 License

This model is published under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE - 2.0).