SGPT 125M Weightedmean Msmarco Specb Bitfit

🚀 SGPT-125M-weightedmean-msmarco-specb-bitfit

このモデルは文の類似度を計算するためのもので、sentence-transformersを利用して特徴抽出を行います。MTEBの様々なタスクで評価されており、多言語の分類、検索、クラスタリングなどのタスクに対応しています。

📚 詳細ドキュメント

モデルの基本情報

評価結果

分類タスク

• MTEB AmazonCounterfactualClassification (en)
  • 正解率: 61.23880597014926
  • 平均適合率: 25.854431650388644
  • F1値: 55.751862762818604
• MTEB AmazonCounterfactualClassification (de)
  • 正解率: 56.88436830835117
  • 平均適合率: 72.67279104379772
  • F1値: 54.449840243786404
• MTEB AmazonCounterfactualClassification (en-ext)
  • 正解率: 58.27586206896551
  • 平均適合率: 14.067357642500387
  • F1値: 48.172318518691334
• MTEB AmazonCounterfactualClassification (ja)
  • 正解率: 54.64668094218415
  • 平均適合率: 11.776694555054965
  • F1値: 44.526622834078765
• MTEB AmazonPolarityClassification
  • 正解率: 65.401225
  • 平均適合率: 60.22809958678552
  • F1値: 65.0251824898292
• MTEB AmazonReviewsClassification (en)
  • 正解率: 31.165999999999993
  • F1値: 30.908870050167437
• MTEB AmazonReviewsClassification (de)
  • 正解率: 24.79
  • F1値: 24.5833598854121
• MTEB AmazonReviewsClassification (es)
  • 正解率: 26.643999999999995
  • F1値: 26.39012792213563
• MTEB AmazonReviewsClassification (fr)
  • 正解率: 26.386000000000003
  • F1値: 26.276867791454873
• MTEB AmazonReviewsClassification (ja)
  • 正解率: 22.078000000000003
  • F1値: 21.797960290226843
• MTEB AmazonReviewsClassification (zh)
  • 正解率: 24.274
  • F1値: 23.887054434822627
• MTEB Banking77Classification
  • 正解率: 77.70454545454545
  • F1値: 77.6929000113803

検索タスク

• MTEB ArguAna
  • MAP@1: 22.404
  • MAP@10: 36.845
  • MAP@100: 37.945
  • MAP@1000: 37.966
  • MAP@3: 31.78
  • MAP@5: 34.608
  • MRR@1: 22.902
  • MRR@10: 37.034
  • MRR@100: 38.134
  • MRR@1000: 38.155
  • MRR@3: 31.935000000000002
  • MRR@5: 34.812
  • NDCG@1: 22.404
  • NDCG@10: 45.425
  • NDCG@100: 50.354
  • NDCG@1000: 50.873999999999995
  • NDCG@3: 34.97
  • NDCG@5: 40.081
  • Precision@1: 22.404
  • Precision@10: 7.303999999999999
  • Precision@100: 0.951
  • Precision@1000: 0.099
  • Precision@3: 14.746
  • Precision@5: 11.337
  • Recall@1: 22.404
  • Recall@10: 73.044
  • Recall@100: 95.092
  • Recall@1000: 99.075
  • Recall@3: 44.239
  • Recall@5: 56.686
• MTEB CQADupstackAndroidRetrieval
  • MAP@1: 22.139
  • MAP@10: 28.839
  • MAP@100: 30.023
  • MAP@1000: 30.153000000000002
  • MAP@3: 26.521
  • MAP@5: 27.775
  • MRR@1: 26.466
  • MRR@10: 33.495000000000005
  • MRR@100: 34.416999999999994
  • MRR@1000: 34.485
  • MRR@3: 31.402
  • MRR@5: 32.496
  • NDCG@1: 26.466
  • NDCG@10: 33.372
  • NDCG@100: 38.7
  • NDCG@1000: 41.696
  • NDCG@3: 29.443
  • NDCG@5: 31.121
  • Precision@1: 26.466
  • Precision@10: 6.037
  • Precision@100: 1.0670000000000002
  • Precision@1000: 0.16199999999999998
  • Precision@3: 13.782
  • Precision@5: 9.757
  • Recall@1: 22.139
  • Recall@10: 42.39
  • Recall@100: 65.427
  • Recall@1000: 86.04899999999999
  • Recall@3: 31.127
  • Recall@5: 35.717999999999996
• MTEB CQADupstackEnglishRetrieval
  • MAP@1: 20.652
  • MAP@10: 27.558
  • MAP@100: 28.473
  • MAP@1000: 28.577
  • MAP@3: 25.402
  • MAP@5: 26.68
  • MRR@1: 25.223000000000003
  • MRR@10: 31.966
  • MRR@100: 32.664
  • MRR@1000: 32.724
  • MRR@3: 30.074
  • MRR@5: 31.249
  • NDCG@1: 25.223000000000003
  • NDCG@10: 31.694
  • NDCG@100: 35.662
  • NDCG@1000: 38.092
  • NDCG@3: 28.294000000000004
  • NDCG@5: 30.049
  • Precision@1: 25.223000000000003
  • Precision@10: 5.777
  • Precision@100: 0.9730000000000001
  • Precision@1000: 0.13999999999999999
  • Precision@3: 13.397
  • Precision@5: 9.605
  • Recall@1: 20.652
  • Recall@10: 39.367999999999995
  • Recall@100: 56.485
  • Recall@1000: 73.292
  • Recall@3: 29.830000000000002
  • Recall@5: 34.43
• MTEB CQADupstackGamingRetrieval
  • MAP@1: 25.180000000000003
  • MAP@10: 34.579
  • MAP@100: 35.589999999999996
  • MAP@1000: 35.68
  • MAP@3: 31.735999999999997
  • MAP@5: 33.479
  • MRR@1: 29.467
  • MRR@10: 37.967
  • MRR@100: 38.800000000000004
  • MRR@1000: 38.858
  • MRR@3: 35.465
  • MRR@5: 37.057
  • NDCG@1: 29.467
  • NDCG@10: 39.796
  • NDCG@100: 44.531
  • NDCG@1000: 46.666000000000004
  • NDCG@3: 34.676
  • NDCG@5: 37.468
  • Precision@1: 29.467
  • Precision@10: 6.601999999999999
  • Precision@100: 0.9900000000000001
  • Precision@1000: 0.124
  • Precision@3: 15.568999999999999
  • Precision@5: 11.172
  • Recall@1: 25.180000000000003
  • Recall@10: 52.269
  • Recall@100: 73.574
  • Recall@1000: 89.141
  • Recall@3: 38.522
  • Recall@5: 45.323
• MTEB CQADupstackGisRetrieval
  • MAP@1: 16.303
  • MAP@10: 21.629
  • MAP@100: 22.387999999999998
  • MAP@1000: 22.489
  • MAP@3: 19.608
  • MAP@5: 20.774
  • MRR@1: 17.740000000000002
  • MRR@10: 23.214000000000002
  • MRR@100: 23.97
  • MRR@1000: 24.054000000000002
  • MRR@3: 21.243000000000002
  • MRR@5: 22.322
  • NDCG@1: 17.740000000000002
  • NDCG@10: 25.113000000000003
  • NDCG@100: 29.287999999999997
  • NDCG@1000: 32.204
  • NDCG@3: 21.111
  • NDCG@5: 23.061999999999998
  • Precision@1: 17.740000000000002
  • Precision@10: 3.955
  • Precision@100: 0.644
  • Precision@1000: 0.093
  • Precision@3: 8.851
  • Precision@5: 6.418
  • Recall@1: 16.303
  • Recall@10: 34.487
  • Recall@100: 54.413999999999994
  • Recall@1000: 77.158
  • Recall@3: 23.733
  • Recall@5: 28.381
• MTEB CQADupstackMathematicaRetrieval
  • MAP@1: 10.133000000000001
  • MAP@10: 15.665999999999999
  • MAP@100: 16.592000000000002
  • MAP@1000: 16.733999999999998
  • MAP@3: 13.625000000000002
  • MAP@5: 14.721
  • MRR@1: 12.562000000000001
  • MRR@10: 18.487000000000002
  • MRR@100: 19.391
  • MRR@1000: 19.487
  • MRR@3: 16.418
  • MRR@5: 17.599999999999998
  • NDCG@1: 12.562000000000001
  • NDCG@10: 19.43
  • NDCG@100: 24.546
  • NDCG@1000: 28.193
  • NDCG@3: 15.509999999999998
  • NDCG@5: 17.322000000000003
  • Precision@1: 12.562000000000001
  • Precision@10: 3.794
  • Precision@100: 0.74
  • Precision@1000: 0.122
  • Precision@3: 7.546
  • Precision@5: 5.721
  • Recall@1: 10.133000000000001
  • Recall@10: 28.261999999999997
  • Recall@100: 51.742999999999995
  • Recall@1000: 78.075
  • Recall@3: 17.634
  • Recall@5: 22.128999999999998
• MTEB CQADupstackPhysicsRetrieval
  • MAP@1: 19.991999999999997
  • MAP@10: 27.346999999999998
  • MAP@100: 28.582
  • MAP@1000: 28.716
  • MAP@3: 24.907
  • MAP@5: 26.1
  • MRR@1: 23.773
  • MRR@10: 31.647
  • MRR@100: 32.639
  • MRR@1000: 32.706
  • MRR@3: 29.195
  • MRR@5: 30.484
  • NDCG@1: 23.773
  • NDCG@10: 32.322
  • NDCG@100: 37.996
  • NDCG@1000: 40.819
  • NDCG@3: 27.876
  • NDCG@5: 29.664
  • Precision@1: 23.773
  • Precision@10: 5.976999999999999
  • Precision@100: 1.055
  • Precision@1000: 0.15
  • Precision@3: 13.122
  • Precision@5: 9.451
  • Recall@1: 19.991999999999997
  • Recall@10: 43.106
  • Recall@100: 67.264
  • Recall@1000: 86.386
  • Recall@3: 30.392000000000003
  • Recall@5: 34.910999999999994
• MTEB CQADupstackProgrammersRetrieval
  • MAP@1: 17.896
  • MAP@10: 24.644
  • MAP@100: 25.790000000000003
  • MAP@1000: 25.913999999999998
  • MAP@3: 22.694
  • MAP@5: 23.69
  • MRR@1: 21.346999999999998
  • MRR@10: 28.594
  • MRR@100: 29.543999999999997
  • MRR@1000: 29.621
  • MRR@3: 26.807
  • MRR@5: 27.669
  • NDCG@1: 21.346999999999998
  • NDCG@10: 28.833
  • NDCG@100: 34.272000000000006
  • NDCG@1000: 37.355
  • NDCG@3: 25.373
  • NDCG@5: 26.756
  • Precision@1: 21.346999999999998
  • Precision@10: 5.2170000000000005
  • Precision@100: 0.954
  • Precision@1000: 0.13899999999999998
  • Precision@3: 11.948
  • Precision@5: 8.425
  • Recall@1: 17.896
  • Recall@10: 37.291000000000004
  • Recall@100: 61.138000000000005
  • Recall@1000: 83.212
  • Recall@3: 27.705999999999996
  • Recall@5: 31.234

クラスタリングタスク

• MTEB ArxivClusteringP2P
  • V-measure: 39.70858340673288
• MTEB ArxivClusteringS2S
  • V-measure: 28.242847713721048
• MTEB BiorxivClusteringP2P
  • V-measure: 33.63260395543984
• MTEB BiorxivClusteringS2S
  • V-measure: 27.038042665369925

再ランキングタスク

• MTEB AskUbuntuDupQuestions
  • MAP: 55.83700395192393
  • MRR: 70.3891307215407

STSタスク

• MTEB BIOSSES
  • コサイン類似度のピアソン相関係数: 79.25366801756223
  • コサイン類似度のスピアマン相関係数: 75.20954502580506
  • ユークリッド距離のピアソン相関係数: 78.79900722991617
  • ユークリッド距離のスピアマン相関係数: 77.79996549607588
  • マンハッタン距離のピアソン相関係数: 78.18408109480399
  • マンハッタン距離のスピアマン相関係数: 76.85958262303106