KoSimCSE-roberta Open-source Korean Sentence Vector Embedding Model

Kosimcse Roberta

Developed by BM-K

A Korean sentence vector embedding model based on the RoBERTa architecture, optimized for sentence representation through contrastive learning, suitable for tasks such as semantic similarity calculation.

Text Embedding

Transformers

Korean#Korean sentence vector embedding #High-precision semantic similarity #Multi-task learning

Downloads 10.35k

Release Time : 4/5/2022

Model Overview

This model uses the RoBERTa architecture for pre-training and optimizes sentence embedding representations through the SimCSE contrastive learning method, capable of generating high-quality Korean sentence vectors for natural language processing tasks such as semantic search and text similarity calculation.

Model Features

Efficient sentence embedding

Optimizes sentence representation through contrastive learning to generate high-quality sentence vectors

Multi-task learning

The multi-task version further enhances performance by combining multiple training objectives

High performance

Achieves SOTA level in Korean semantic similarity tasks

Model Capabilities

Sentence embedding generation

Semantic similarity calculation

Text retrieval

Sentence clustering

Use Cases

Information retrieval

Semantic search

Using sentence vectors for similar document retrieval

Obtains more relevant results compared to traditional keyword search

Text analysis

Text similarity calculation

Calculating semantic similarity between two Korean sentences

Achieves an average score of 85.77 on the test set

🚀 Korean-Sentence-Embedding

This repository is dedicated to Korean sentence embedding. You can download pre-trained models and start inference immediately. It also offers an environment for individual model training.

🚀 Quick Start

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoModel, AutoTokenizer

def cal_score(a, b):
    if len(a.shape) == 1: a = a.unsqueeze(0)
    if len(b.shape) == 1: b = b.unsqueeze(0)

    a_norm = a / a.norm(dim=1)[:, None]
    b_norm = b / b.norm(dim=1)[:, None]
    return torch.mm(a_norm, b_norm.transpose(0, 1)) * 100

model = AutoModel.from_pretrained('BM-K/KoSimCSE-roberta')
tokenizer = AutoTokenizer.from_pretrained('BM-K/KoSimCSE-roberta')

sentences = ['치타가 들판을 가로 질러 먹이를 쫓는다.',
             '치타 한 마리가 먹이 뒤에서 달리고 있다.',
             '원숭이 한 마리가 드럼을 연주한다.']

inputs = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt")
embeddings, _ = model(**inputs, return_dict=False)

score01 = cal_score(embeddings[0][0], embeddings[1][0])
score02 = cal_score(embeddings[0][0], embeddings[2][0])

📊 Performance

Semantic Textual Similarity test set results

Property	Details
Model Link	https://github.com/BM-K/Sentence-Embedding-is-all-you-need

Model	AVG	Cosine Pearson	Cosine Spearman	Euclidean Pearson	Euclidean Spearman	Manhattan Pearson	Manhattan Spearman	Dot Pearson	Dot Spearman
KoSBERT^†_SKT	77.40	78.81	78.47	77.68	77.78	77.71	77.83	75.75	75.22
KoSBERT	80.39	82.13	82.25	80.67	80.75	80.69	80.78	77.96	77.90
KoSRoBERTa	81.64	81.20	82.20	81.79	82.34	81.59	82.20	80.62	81.25

KoSentenceBART	77.14	79.71	78.74	78.42	78.02	78.40	78.00	74.24	72.15
KoSentenceT5	77.83	80.87	79.74	80.24	79.36	80.19	79.27	72.81	70.17

KoSimCSE-BERT^†_SKT	81.32	82.12	82.56	81.84	81.63	81.99	81.74	79.55	79.19
KoSimCSE-BERT	83.37	83.22	83.58	83.24	83.60	83.15	83.54	83.13	83.49
KoSimCSE-RoBERTa	83.65	83.60	83.77	83.54	83.76	83.55	83.77	83.55	83.64

KoSimCSE-BERT-multitask	85.71	85.29	86.02	85.63	86.01	85.57	85.97	85.26	85.93
KoSimCSE-RoBERTa-multitask	85.77	85.08	86.12	85.84	86.12	85.83	86.12	85.03	85.99

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご