All Mpnet Base V2

🚀 all-mpnet-base-v2

This is a sentence-transformers model that maps sentences & paragraphs to a 768-dimensional dense vector space, useful for tasks like clustering or semantic search.

🚀 Quick Start

This model can be used easily with sentence-transformers installed. First, install the library:

pip install -U sentence-transformers

Then, you can use the model as follows:

from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')
embeddings = model.encode(sentences)
print(embeddings)

✨ Features

• Maps sentences and paragraphs to a 768-dimensional dense vector space.
• Can be used for clustering, semantic search, and other related tasks.

📦 Installation

To use this model, you need to install the sentence-transformers library:

pip install -U sentence-transformers

💻 Usage Examples

Basic Usage

If you have sentence-transformers installed, using the model is straightforward:

from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')
embeddings = model.encode(sentences)
print(embeddings)

Advanced Usage

Without sentence-transformers, you can use the model by passing the input through the transformer model and applying the right pooling operation on the contextualized word embeddings:

from transformers import AutoTokenizer, AutoModel
import torch
import torch.nn.functional as F

#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)


# Sentences we want sentence embeddings for
sentences = ['This is an example sentence', 'Each sentence is converted']

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-mpnet-base-v2')
model = AutoModel.from_pretrained('sentence-transformers/all-mpnet-base-v2')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

# Normalize embeddings
sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1)

print("Sentence embeddings:")
print(sentence_embeddings)

📚 Documentation

Evaluation Results

For an automated evaluation of this model, see the Sentence Embeddings Benchmark: https://seb.sbert.net

Background

The project aims to train sentence embedding models on very large sentence-level datasets using a self-supervised contrastive learning objective. The pretrained microsoft/mpnet-base model was used and fine-tuned on a 1B sentence pairs dataset. A contrastive learning objective was applied, where given a sentence from a pair, the model should predict which out of a set of randomly sampled other sentences was actually paired with it in the dataset.

This model was developed during the Community week using JAX/Flax for NLP & CV organized by Hugging Face, as part of the project Train the Best Sentence Embedding Model Ever with 1B Training Pairs. Efficient hardware infrastructure (7 TPUs v3-8) was used, and there was intervention from Google's Flax, JAX, and Cloud team members regarding efficient deep learning frameworks.

Intended uses

This model is intended to be used as a sentence and short paragraph encoder. Given an input text, it outputs a vector that captures the semantic information. The sentence vector can be used for information retrieval, clustering, or sentence similarity tasks. By default, input text longer than 384 word pieces is truncated.

Training procedure

Pre-training

The pretrained microsoft/mpnet-base model was used. Refer to the model card for more detailed information about the pre-training procedure.

Fine-tuning

The model was fine-tuned using a contrastive objective. Formally, the cosine similarity was computed for each possible sentence pair from the batch, and then the cross-entropy loss was applied by comparing with the true pairs.

Hyper parameters

The model was trained on a TPU v3-8 for 100k steps with a batch size of 1024 (128 per TPU core). A learning rate warm-up of 500 was used. The sequence length was limited to 128 tokens. The AdamW optimizer with a 2e-5 learning rate was used. The full training script is available in the current repository: train_script.py.

Training data

Multiple datasets were concatenated to fine-tune the model. The total number of sentence pairs is over 1 billion. Each dataset was sampled with a weighted probability, and the configuration is detailed in the data_config.json file.

📄 License

This project is licensed under the Apache-2.0 license.