The open-source sgpt-bloom-1b7-nli model - Free deployment to achieve sentence similarity and inference judgment

Sgpt Bloom 1b7 Nli

Developed by bigscience-data

A sentence transformer trained on the BLOOM-1.7B model, specializing in sentence similarity and natural language inference tasks

Text Embedding

Safetensors

#Multilingual semantic search #Weighted mean pooling #NLI fine-tuning

Downloads 14

Release Time : 5/29/2022

Model Overview

This model is a sentence embedding model based on the BLOOM architecture, specifically designed for computing sentence similarity and performing natural language inference tasks. It employs techniques like weighted mean pooling to optimize semantic representation capabilities.

Model Features

Weighted mean pooling

Utilizes weighted mean pooling technology to optimize sentence representation and enhance semantic similarity computation effectiveness

BitFit training

An efficient training method that only trains bias parameters, reducing computational resource requirements while maintaining performance

Gradient caching

Employs gradient caching technology to handle large batch training, improving training efficiency

Multilingual support

Performs excellently in multiple languages including Chinese and French

Model Capabilities

Sentence similarity computation

Semantic textual similarity evaluation

Text classification

Cross-lingual semantic matching

Use Cases

E-commerce

Product review classification

Classifying sentiment or topics of Amazon product reviews

French review classification accuracy 39.29%, Chinese 37.63%

Dialogue systems

Intent recognition

Identifying intent categories in user dialogues

French intent classification accuracy 63.36%

Scenario classification

Identifying the scenario category of user dialogues

French scenario classification accuracy 69.60%

Semantic search

Cross-lingual semantic matching

Computing semantic similarity between texts in different languages

Chinese STS task Pearson correlation coefficient 59.72, French 73.44

🚀 sgpt-bloom-1b7-nli

This model is designed for sentence similarity tasks, offering feature extraction capabilities and achieving certain performance in multiple classification and STS tasks.

🚀 Quick Start

For usage instructions, refer to: https://github.com/Muennighoff/sgpt#symmetric-semantic-search

The model was trained with the command

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch examples/training/nli/training_nli_v2.py --model_name bigscience/bloom-1b3 --freezenonbias --train_batch_size 128 --lr 32e-5 --pooling weightedmean --wandb --wandbwatchlog gradients --gradcache --chunksize 4

✨ Features

Sentence Similarity: Specialized in calculating sentence similarity.
Feature Extraction: Capable of extracting features from sentences.
Multi - language Support: Demonstrates performance in multiple languages such as French and Chinese.

📚 Documentation

Model Information

Property	Details
Pipeline Tag	sentence-similarity
Tags	sentence-transformers, feature-extraction, sentence-similarity, mteb

Model Index

The model sgpt-bloom-1b7-nli has the following evaluation results on different datasets:

Task Type	Dataset	Language	Accuracy	F1	Other Metrics
Classification	mteb/amazon_reviews_multi	fr	39.286	38.87078070073539	-
Classification	mteb/amazon_reviews_multi	zh	37.634	36.86046604093418	-
Classification	mteb/mtop_domain	fr	83.79893517068588	83.72326662566203	-
Classification	mteb/mtop_intent	fr	63.36047604134043	44.261707019308126	-
Classification	mteb/amazon_massive_intent	fr	64.57632817753867	62.60453982786661	-
Classification	mteb/amazon_massive_scenario	fr	69.59986550100874	69.71803697939914	-
STS	mteb/sts22-crosslingual-sts	zh	-	-	cos_sim_pearson: 59.71781185663265 cos_sim_spearman: 58.538648447630514 euclidean_pearson: 53.53848180206165 euclidean_spearman: 56.33730262964236 manhattan_pearson: 54.62109820575505 manhattan_spearman: 57.223846291318914
STS	mteb/sts22-crosslingual-sts	fr	-	-	cos_sim_pearson: 73.44021434651606 cos_sim_spearman: 73.13412769502769 euclidean_pearson: 68.16368597409867 euclidean_spearman: 72.44964781564485 manhattan_pearson: 69.42307032478939 manhattan_spearman: 73.3523195012387

Evaluation Results

{'askubuntu': 57.44, 'cqadupstack': 14.18, 'twitterpara': 73.99, 'scidocs': 74.74, 'avg': 55.087500000000006}

Training

The model was trained with the following parameters:

DataLoader:

sentence_transformers.datasets.NoDuplicatesDataLoader.NoDuplicatesDataLoader of length 4403 with parameters:

{'batch_size': 128}

The model uses BitFit, weighted-mean pooling & GradCache, for details see: https://arxiv.org/abs/2202.08904

Loss:

sentence_transformers.losses.MultipleNegativesRankingLoss.MNRLGradCache

Parameters of the fit()-Method:

{
    "epochs": 1,
    "evaluation_steps": 440,
    "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
    "max_grad_norm": 1,
    "optimizer_class": "<class 'transformers.optimization.AdamW'>",
    "optimizer_params": {
        "lr": 0.00032
    },
    "scheduler": "WarmupLinear",
    "steps_per_epoch": null,
    "warmup_steps": 441,
    "weight_decay": 0.01
}

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 75, 'do_lower_case': False}) with Transformer model: BloomModel 
  (1): Pooling({'word_embedding_dimension': 2048, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False})
)

Citing & Authors

@article{muennighoff2022sgpt,
  title={SGPT: GPT Sentence Embeddings for Semantic Search},
  author={Muennighoff, Niklas},
  journal={arXiv preprint arXiv:2202.08904},
  year={2022}
}

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご