Simcse Ja Bert Base Clcmlp

🚀 Japanese SimCSE (BERT-base)

A Japanese SimCSE model for easily extracting sentence embedding representations from Japanese sentences.

🚀 Quick Start

This is a Japanese SimCSE model, named pkshatech/simcse-ja-bert-base-clcmlp. It allows you to effortlessly extract sentence embedding representations from Japanese sentences. The model is based on cl-tohoku/bert-base-japanese-v2 and trained on the JSNLI dataset, a Japanese natural language inference dataset.

Japanese README

✨ Features

• Extract sentence embedding representations from Japanese sentences.
• Based on a well - known Japanese BERT model and trained on a Japanese NLI dataset.

📦 Installation

You can use this model easily with sentence-transformers. For tokenization, you need fugashi and unidic-lite.

Install sentence-transformers, fugashi, and unidic-lite with pip as follows:

pip install -U fugashi[unidic-lite] sentence-transformers

💻 Usage Examples

Basic Usage

You can load the model and convert sentences to dense vectors as follows:

from sentence_transformers import SentenceTransformer
sentences = [
    "PKSHA Technologyは機械学習/深層学習技術に関わるアルゴリズムソリューションを展開している。",
    "この深層学習モデルはPKSHA Technologyによって学習され、公開された。",
    "広目天は、仏教における四天王の一尊であり、サンスクリット語の「種々の眼をした者」を名前の由来とする。",
]

model = SentenceTransformer('pkshatech/simcse-ja-bert-base-clcmlp')
embeddings = model.encode(sentences)
print(embeddings)

Since the loss function used during training is cosine similarity, we recommend using cosine similarity for downstream tasks.

🔧 Technical Details

Tokenization

We use the same tokenizer as tohoku/bert-base-japanese-v2. For details, please see the README of tohoku/bert-base-japanese-v2.

Training

We set tohoku/bert-base-japanese-v2 as the initial value and trained it on the train set of JSNLI. We trained for 20 epochs and published the checkpoint of the model with the highest Spearman's correlation coefficient on the validation set [^1] of the train set of JSTS.

Training Parameters

📄 License

This model is distributed under the terms of the Creative Commons Attribution - ShareAlike 4.0.

[^1]: When we trained this model, the test data of JGLUE was not released, so we used the dev set of JGLUE as a private evaluation data. Therefore, we selected the checkpoint on the train set of JGLUE instead of its dev set.