SGPT-5.8B Open Source Model - Designed for Sentence Similarity Calculation and Feature Extraction, Free to Deploy!

SGPT 5.8B Weightedmean Nli Bitfit

Developed by Muennighoff

SGPT-5.8B is a sentence transformer model based on a 5.8B parameter scale, optimized through weighted mean and NLI (Natural Language Inference) fine-tuning, specifically designed for sentence similarity calculation and feature extraction tasks.

Text Embedding

PyTorch

#Multilingual sentence similarity #NLI fine-tuning #Large-scale feature extraction

Downloads 86

Release Time : 3/2/2022

Model Overview

This model is primarily used for sentence similarity calculation and feature extraction, excelling in multiple tasks on the MTEB (Massive Text Embedding Benchmark), supporting multiple languages and cross-lingual tasks.

Model Features

Multilingual Support

Supports sentence similarity calculation and feature extraction in multiple languages, including English, German, French, Spanish, Japanese, and Chinese.

High Performance

Performs excellently in multiple MTEB tasks, especially achieving high scores in sentence similarity and classification tasks.

NLI Fine-tuning

Fine-tuned through Natural Language Inference (NLI) tasks, enhancing the model's capabilities in sentence similarity and semantic understanding.

Model Capabilities

Sentence similarity calculation

Feature extraction

Text classification

Cross-lingual text mining

Semantic retrieval

Use Cases

E-commerce

Product review classification

Used for sentiment analysis and classification of product reviews on e-commerce platforms like Amazon.

Achieved 82.31% accuracy in the MTEB Amazon polarity classification task.

Information Retrieval

Q&A systems

Used for similar question retrieval and answer matching in Q&A systems.

Achieved an average precision of 55.90 in the AskUbuntu duplicate question task.

Cross-lingual Applications

Bilingual text mining

Used for cross-lingual text matching and alignment tasks.

Achieved 75.49% accuracy in the BUCC German-English task.

🚀 SGPT-5.8B-weightedmean-nli-bitfit

This model is designed for sentence - similarity tasks. It offers feature - extraction capabilities and has been evaluated on various datasets from the MTEB benchmark.

📚 Documentation

Model Information

Property	Details
Pipeline Tag	sentence - similarity
Tags	sentence - transformers, feature - extraction, sentence - similarity, mteb
Model Name	SGPT - 5.8B - weightedmean - nli - bitfit

Evaluation Results

Classification Tasks

Task	Dataset	Language	Accuracy	AP	F1
Classification	MTEB AmazonCounterfactualClassification	en	74.07462686567165	37.44692407529112	68.28971003916419
Classification	MTEB AmazonCounterfactualClassification	de	66.63811563169165	78.57252079915924	64.5543087846584
Classification	MTEB AmazonCounterfactualClassification	en - ext	77.21889055472263	25.663426367826712	64.26265688503176
Classification	MTEB AmazonCounterfactualClassification	ja	58.06209850107067	14.028219107023915	48.10387189660778
Classification	MTEB AmazonPolarityClassification	default	82.30920000000002	76.88786578621213	82.15455656065011
Classification	MTEB AmazonReviewsClassification	en	41.584	-	41.203137944390114
Classification	MTEB AmazonReviewsClassification	de	35.288000000000004	-	34.672995558518096
Classification	MTEB AmazonReviewsClassification	es	38.34	-	37.608755629529455
Classification	MTEB AmazonReviewsClassification	fr	37.839999999999996	-	36.86898201563507
Classification	MTEB AmazonReviewsClassification	ja	30.936000000000003	-	30.49401738527071
Classification	MTEB AmazonReviewsClassification	zh	33.75	-	33.38338946025617
Classification	MTEB Banking77Classification	default	81.74025974025973	-	81.66568824876

Retrieval Tasks

Task	Dataset	MAP@1	MAP@10	MAP@100	MAP@1000	MAP@3	MAP@5	NDCG@1	NDCG@10	NDCG@100	NDCG@1000	NDCG@3	NDCG@5	Precision@1	Precision@10	Precision@100	Precision@1000	Precision@3	Precision@5	Recall@1	Recall@10	Recall@100	Recall@1000	Recall@3	Recall@5
Retrieval	MTEB ArguAna	13.727	26.740000000000002	28.218	28.246	21.728	24.371000000000002	13.727	35.07	41.947	42.649	24.484	29.282999999999998	13.727	6.223	0.9369999999999999	0.099	10.835	8.848	13.727	62.233000000000004	93.67	99.14699999999999	32.504	44.239
Retrieval	MTEB CQADupstackAndroidRetrieval	26.715	35.007	36.352000000000004	36.51	32.257999999999996	33.595000000000006	33.906	40.353	45.562999999999995	48.454	36.349	37.856	33.906	7.854	1.29	0.188	17.549	12.561	26.715	49.508	71.76599999999999	91.118	37.356	41.836
Retrieval	MTEB CQADupstackEnglishRetrieval	19.663	27.086	28.066999999999997	28.18	24.819	26.332	25.732	31.613999999999997	35.757	38.21	28.332	30.264000000000003	25.732	6.038	1.034	0.149	13.864	10.241999999999999	19.663	39.585	57.718	74.26700000000001	29.845	35.105
Retrieval	MTEB CQADupstackGamingRetrieval	30.125	39.824	40.935	41.019	37.144	38.647999999999996	34.922	45.072	50.046	51.895	40.251	-	34.922	-	-	-	-	-	30.125	-	-	-	-	-

Clustering Tasks

Task	Dataset	V - Measure
Clustering	MTEB ArxivClusteringP2P	40.553923271901695
Clustering	MTEB ArxivClusteringS2S	32.49323183712211
Clustering	MTEB BiorxivClusteringP2P	33.59451202614059
Clustering	MTEB BiorxivClusteringS2S	29.128241446157165

Reranking Task

Task	Dataset	MAP	MRR
Reranking	MTEB AskUbuntuDupQuestions	55.89811361443445	70.16235764850724

STS Task

Task	Dataset	Cos Sim Pearson	Cos Sim Spearman	Euclidean Pearson	Euclidean Spearman	Manhattan Pearson	Manhattan Spearman
STS	MTEB BIOSSES	82.50506557805856	79.50000423261176	75.76190885392926	76.7330737163434	75.825318036112	76.7415076434559

Bitext Mining Tasks

Task	Dataset	Language Pair	Accuracy	F1	Precision	Recall
BitextMining	MTEB BUCC	de - en	75.49060542797494	75.15379262352123	74.99391092553932	75.49060542797494
BitextMining	MTEB BUCC	fr - en	0.4182258419546555	0.4182258419546555	0.4182258419546555	0.4182258419546555
BitextMining	MTEB BUCC	ru - en	0.013855213023900243	0.0115460108532502	0.010391409767925183	0.013855213023900243
BitextMining	MTEB BUCC	zh - en	0.315955766192733	0.315955766192733	0.315955766192733	0.315955766192733

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご