All Mpnet Base V2

🚀 All MPNet base model (v2) for Semantic Search

This model maps sentences & paragraphs to a 768-dimensional dense vector space, facilitating tasks like clustering or semantic search.

🚀 Quick Start

This section provides two ways to use the model: with sentence-transformers and with HuggingFace Transformers.

✨ Features

• Maps sentences and paragraphs to a 768-dimensional dense vector space.
• Suitable for tasks such as clustering and semantic search.
• Can be used as a sentence and short paragraph encoder.

📦 Installation

To use this model, you need to install sentence-transformers if you haven't already:

pip install -U sentence-transformers

💻 Usage Examples

Basic Usage

If you have sentence-transformers installed, using this model is straightforward:

from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')
embeddings = model.encode(sentences)
print(embeddings)

Advanced Usage

Without sentence-transformers, you can use the model as follows. First, pass your input through the transformer model, then apply the appropriate pooling operation on top of the contextualized word embeddings.

from transformers import AutoTokenizer, AutoModel
import torch
import torch.nn.functional as F

#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)


# Sentences we want sentence embeddings for
sentences = ['This is an example sentence', 'Each sentence is converted']

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-mpnet-base-v2')
model = AutoModel.from_pretrained('sentence-transformers/all-mpnet-base-v2')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

# Normalize embeddings
sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1)

print("Sentence embeddings:")
print(sentence_embeddings)

📚 Documentation

Evaluation Results

For an automated evaluation of this model, see the Sentence Embeddings Benchmark: https://seb.sbert.net

Background

The project aims to train sentence embedding models on very large sentence-level datasets using a self-supervised contrastive learning objective. The pretrained microsoft/mpnet-base model was used and fine-tuned on a 1B sentence pairs dataset. The model was developed during the Community week using JAX/Flax for NLP & CV, organized by Hugging Face, as part of the project Train the Best Sentence Embedding Model Ever with 1B Training Pairs.

Intended uses

This model is intended to be used as a sentence and short paragraph encoder. Given an input text, it outputs a vector that captures the semantic information. The sentence vector can be used for information retrieval, clustering, or sentence similarity tasks. By default, input text longer than 384 word pieces is truncated.

Training procedure

Pre-training

The pretrained microsoft/mpnet-base model is used. Refer to the model card for more detailed information about the pre-training procedure.

Fine-tuning

The model is fine-tuned using a contrastive objective. Formally, the cosine similarity is computed for each possible sentence pair from the batch, and then the cross-entropy loss is applied by comparing with the true pairs.

Hyper parameters

The model was trained on a TPU v3-8 for 100k steps with a batch size of 1024 (128 per TPU core). A learning rate warm-up of 500 was used, the sequence length was limited to 128 tokens, and the AdamW optimizer with a 2e-5 learning rate was employed.

Training data

Multiple datasets were concatenated to fine-tune the model, with a total of over 1 billion sentence pairs. Each dataset was sampled with a weighted probability, and the configuration is detailed in the data_config.json file.

📄 License

This project is licensed under the MIT license.