SFR-Embedding-Mistral Open-Source Text Embedding Model - Free Deployment to Boost Text Retrieval Tasks

SFR Embedding Mistral

Developed by Salesforce

A text embedding model developed by Salesforce Research, trained on E5-mistral-7b-instruct and Mistral-7B-v0.1, primarily used for text retrieval tasks.

Text Embedding

Transformers

English#Long Text Embedding #Retrieval-Augmented Generation #Instruction Fine-tuning

Downloads 34.75k

Release Time : 1/24/2024

Model Overview

This model is a text embedding model based on the Mistral architecture, focusing on improving performance in text retrieval tasks. Enhanced through transfer learning techniques, it is suitable for various information retrieval scenarios.

Model Features

Efficient Retrieval Capability

Optimized for text retrieval tasks, capable of efficiently matching queries with relevant documents

Long Text Processing

Supports text inputs of up to 4096 tokens, suitable for processing longer documents

Transfer Learning Enhancement

Trained on E5-mistral-7b-instruct and Mistral-7B-v0.1, delivering superior performance

Model Capabilities

Text Embedding Generation

Semantic Similarity Calculation

Information Retrieval

Document Matching

Use Cases

Information Retrieval

Web Search

Match user queries with webpage content to return the most relevant results

In the example, high matching scores between queries and relevant paragraphs are displayed

Knowledge Base Q&A

Retrieve the most relevant answer passages from a knowledge base based on questions

Content Recommendation

🚀 SFR-Embedding-Mistral

This model, SFR-Embedding-Mistral, is evaluated on multiple tasks in the MTEB benchmark, including Classification, Retrieval, Clustering, Reranking, and STS. It shows different performance metrics on various datasets, which can provide references for related NLP tasks.

📚 Documentation

Model Performance Metrics

The following table presents the performance metrics of the SFR-Embedding-Mistral model on different tasks and datasets:

Task Type	Dataset Name	Accuracy	AP	F1	NDCG@k	MAP@k	Recall@k	Precision@k	MRR@k	V-Measure	Cosine Similarity (Pearson)	Cosine Similarity (Spearman)	Euclidean (Pearson)	Euclidean (Spearman)	Manhattan (Pearson)	Manhattan (Spearman)
Classification	MTEB AmazonCounterfactualClassification (en)	77.92537313432834	40.86767661556651	71.65758897929837	-	-	-	-	-	-	-	-	-	-	-	-
Classification	MTEB AmazonPolarityClassification	95.967	94.46300829592593	95.96507173189292	-	-	-	-	-	-	-	-	-	-	-	-
Classification	MTEB AmazonReviewsClassification (en)	54.352000000000004	-	53.636682615380174	-	-	-	-	-	-	-	-	-	-	-	-
Retrieval	MTEB ArguAna	-	-	-	See detailed values for different k	See detailed values for different k	See detailed values for different k	See detailed values for different k	See detailed values for different k	-	-	-	-	-	-	-
Clustering	MTEB ArxivClusteringP2P	-	-	-	-	-	-	-	-	52.07508593014336	-	-	-	-	-	-
Clustering	MTEB ArxivClusteringS2S	-	-	-	-	-	-	-	-	47.381339333240675	-	-	-	-	-	-
Reranking	MTEB AskUbuntuDupQuestions	-	-	-	-	67.58376647859171	-	-	80.56885635140483	-	-	-	-	-	-	-
STS	MTEB BIOSSES	-	-	-	-	-	-	-	-	-	88.40107280274783	86.07003345325681	87.1726034325395	86.07003345325681	87.25660625029772	86.3808839096893
Classification	MTEB Banking77Classification	88.81168831168831	-	88.76514496560141	-	-	-	-	-	-	-	-	-	-	-	-
Clustering	MTEB BiorxivClusteringP2P	-	-	-	-	-	-	-	-	43.9382520874344	-	-	-	-	-	-
Clustering	MTEB BiorxivClusteringS2S	-	-	-	-	-	-	-	-	41.14351847240913	-	-	-	-	-	-
Retrieval	MTEB CQADupstackRetrieval	-	-	-	See detailed values for different k	See detailed values for different k	See detailed values for different k	See detailed values for different k	See detailed values for different k	-	-	-	-	-	-	-
Retrieval	MTEB ClimateFEVER	-	-	-	See detailed values for different k	See detailed values for different k	-	-	-	-	-	-	-	-	-	-

Detailed Metrics for Retrieval Tasks

MTEB ArguAna

Metric	Value
NDCG@1	43.314
NDCG@2	54.757
NDCG@3	58.84700000000001
NDCG@5	63.634
NDCG@7	65.741
NDCG@10	67.171
NDCG@20	68.585
NDCG@30	68.81
NDCG@50	68.932
NDCG@70	68.992
NDCG@100	69.014
NDCG@200	69.014
NDCG@300	69.014
NDCG@500	69.014
NDCG@700	69.014
NDCG@1000	69.014
MAP@1	43.314
MAP@2	52.383
MAP@3	55.108999999999995
MAP@5	57.772999999999996
MAP@7	58.718
MAP@10	59.256
MAP@20	59.668
MAP@30	59.709999999999994
MAP@50	59.727
MAP@70	59.733999999999995
MAP@100	59.73500000000001
MAP@200	59.73500000000001
MAP@300	59.73500000000001
MAP@500	59.73500000000001
MAP@700	59.73500000000001
MAP@1000	59.73500000000001
Recall@1	43.314
Recall@2	61.451
Recall@3	69.63000000000001
Recall@5	81.223
Recall@7	87.33999999999999
Recall@10	92.034
Recall@20	97.44
Recall@30	98.506
Recall@50	99.14699999999999
Recall@70	99.502
Recall@100	99.644
Recall@200	99.644
Recall@300	99.644
Recall@500	99.644
Recall@700	99.644
Recall@1000	99.644
Precision@1	43.314
Precision@2	30.725
Precision@3	23.21
Precision@5	16.245
Precision@7	12.477
Precision@10	9.203
Precision@20	4.872
Precision@30	3.2840000000000003
Precision@50	1.983
Precision@70	1.421
Precision@100	0.996
Precision@200	0.498
Precision@300	0.332
Precision@500	0.199
Precision@700	0.14200000000000002
Precision@1000	0.1
MRR@1	44.666
MRR@2	52.418
MRR@3	55.595000000000006
MRR@5	58.205
MRR@7	59.202999999999996
MRR@10	59.727
MRR@20	60.133
MRR@30	60.178
MRR@50	60.192
MRR@70	60.19799999999999
MRR@100	60.199999999999996
MRR@200	60.199999999999996
MRR@300	60.199999999999996
MRR@500	60.199999999999996
MRR@700	60.199999999999996
MRR@1000	60.199999999999996

MTEB CQADupstackRetrieval

Metric	Value
NDCG@1	34.51166666666667
NDCG@2	38.51591666666667
NDCG@3	40.95083333333333
NDCG@5	43.580666666666666
NDCG@7	45.0625
NDCG@10	46.49083333333333
NDCG@20	48.731333333333325
NDCG@30	49.78666666666667
NDCG@50	50.84049999999999
NDCG@70	51.393750000000004
NDCG@100	51.883333333333326
NDCG@200	52.65225
NDCG@300	52.98241666666669
NDCG@500	53.28541666666668
NDCG@700	53.49241666666668
NDCG@1000	53.63758333333334
MAP@1	29.10075
MAP@2	34.636500000000005
MAP@3	36.92033333333333
MAP@5	38.81641666666666
MAP@7	39.635416666666664
MAP@10	40.294583333333335
MAP@20	41.07574999999999
MAP@30	41.333
MAP@50	41.529333333333334
MAP@70	41.606833333333334
MAP@100	41.66224999999999
MAP@200	41.72691666666666
MAP@300	41.746583333333334
MAP@500	41.75983333333333
MAP@700	41.76558333333333
MAP@1000	41.769000000000005
Recall@1	29.10075
Recall@2	39.07658333333333
Recall@3	44.93591666666667
Recall@5	51.66883333333333
Recall@7	55.881000000000014
Recall@10	60.34691666666667
Recall@20	68.44016666666667
Recall@30	72.90766666666667
Recall@50	77.843
Recall@70	80.70366666666668
Recall@100	83.42866666666667
Recall@200	88.06816666666668
Recall@300	90.249
Recall@500	92.37616666666668
Recall@700	93.978
Recall@1000	95.12791666666666
Precision@1	34.51166666666667
Precision@2	24.326333333333327
Precision@3	19.099249999999998
Precision@5	13.672666666666666
Precision@7	10.772
Precision@10	8.302166666666668
Precision@20	4.8960833333333325
Precision@30	3.551083333333333
Precision@50	2.3386666666666662
Precision@70	1.7605833333333334
Precision@100	1.2965
Precision@200	0.7106666666666668
Precision@300	0.4955
Precision@500	0.3106666666666667
Precision@700	0.22791666666666668
Precision@1000	0.1635833333333333
MRR@1	34.51166666666667
MRR@2	39.954249999999995
MRR@3	41.93741666666668
MRR@5	43.487166666666674
MRR@7	44.14983333333333
MRR@10	44.62766666666666
MRR@20	45.15291666666668
MRR@30	45.317
MRR@50	45.42875
MRR@70	45.46966666666667
MRR@100	45.49716666666667
MRR@200	45.525166666666664
MRR@300	45.53233333333335
MRR@500	45.5365
MRR@700	45.538583333333335
MRR@1000	45.539583333333326

MTEB ClimateFEVER

Metric	Value
NDCG@1	35.179
NDCG@2	31.243
NDCG@3	30.562
NDCG@5	32.409
NDCG@7	34.525
NDCG@10	36.415
NDCG@20	39.443
NDCG@30	40.796
NDCG@50	42.16
NDCG@70	42.971
NDCG@100	43.691
NDCG@200	45.004
NDCG@300	45.527
NDCG@500	46.072
NDCG@700	46.387
NDCG@1000	46.663
MAP@1	15.692
MAP@2	20.116
...	...

These metrics comprehensively reflect the performance of the SFR-Embedding-Mistral model on different NLP tasks, which can help users understand the model's capabilities and make appropriate choices.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご