Multi Qa Mpnet Base Dot V1

🚀 multi-qa-mpnet-base-dot-v1

This is a sentence-transformers model designed for semantic search, mapping sentences and paragraphs to a 768-dimensional dense vector space. It's trained on 215M (question, answer) pairs from various sources.

🚀 Quick Start

This model can be used in two ways, with or without the sentence-transformers library.

✨ Features

• Maps sentences & paragraphs to a 768-dimensional dense vector space.
• Designed for semantic search.
• Trained on 215M (question, answer) pairs from diverse sources.

📦 Installation

If you want to use the sentence-transformers way, you need to install the library first:

pip install -U sentence-transformers

💻 Usage Examples

Basic Usage (Sentence-Transformers)

from sentence_transformers import SentenceTransformer, util

query = "How many people live in London?"
docs = ["Around 9 Million people live in London", "London is known for its financial district"]

#Load the model
model = SentenceTransformer('sentence-transformers/multi-qa-mpnet-base-dot-v1')

#Encode query and documents
query_emb = model.encode(query)
doc_emb = model.encode(docs)

#Compute dot score between query and all document embeddings
scores = util.dot_score(query_emb, doc_emb)[0].cpu().tolist()

#Combine docs & scores
doc_score_pairs = list(zip(docs, scores))

#Sort by decreasing score
doc_score_pairs = sorted(doc_score_pairs, key=lambda x: x[1], reverse=True)

#Output passages & scores
for doc, score in doc_score_pairs:
    print(score, doc)

Advanced Usage (HuggingFace Transformers)

from transformers import AutoTokenizer, AutoModel
import torch

#CLS Pooling - Take output from first token
def cls_pooling(model_output):
    return model_output.last_hidden_state[:,0]

#Encode text
def encode(texts):
    # Tokenize sentences
    encoded_input = tokenizer(texts, padding=True, truncation=True, return_tensors='pt')

    # Compute token embeddings
    with torch.no_grad():
        model_output = model(**encoded_input, return_dict=True)

    # Perform pooling
    embeddings = cls_pooling(model_output)

    return embeddings


# Sentences we want sentence embeddings for
query = "How many people live in London?"
docs = ["Around 9 Million people live in London", "London is known for its financial district"]

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/multi-qa-mpnet-base-dot-v1")
model = AutoModel.from_pretrained("sentence-transformers/multi-qa-mpnet-base-dot-v1")

#Encode query and docs
query_emb = encode(query)
doc_emb = encode(docs)

#Compute dot score between query and all document embeddings
scores = torch.mm(query_emb, doc_emb.transpose(0, 1))[0].cpu().tolist()

#Combine docs & scores
doc_score_pairs = list(zip(docs, scores))

#Sort by decreasing score
doc_score_pairs = sorted(doc_score_pairs, key=lambda x: x[1], reverse=True)

#Output passages & scores
for doc, score in doc_score_pairs:
    print(score, doc)

🔧 Technical Details

📚 Documentation

Background

The project aims to train sentence embedding models on very large sentence level datasets using a self-supervised contrastive learning objective. The model should predict which out of a set of randomly sampled other sentences, was actually paired with a given sentence in the dataset.

This model was developed during the Community week using JAX/Flax for NLP & CV, organized by Hugging Face, as part of the project Train the Best Sentence Embedding Model Ever with 1B Training Pairs. The project benefited from efficient hardware infrastructure (7 TPUs v3-8) and the intervention from Google's Flax, JAX, and Cloud team members about efficient deep learning frameworks.

Intended uses

Our model is intended to be used for semantic search. It encodes queries / questions and text paragraphs in a dense vector space and finds relevant documents for the given passages.

⚠️ Important Note

There is a limit of 512 word pieces. Text longer than that will be truncated. Also, the model was just trained on input text up to 250 word pieces, so it might not work well for longer text.

Training procedure

The full training script is accessible in this current repository: train_script.py.

Pre-training

We use the pretrained mpnet-base model. Please refer to the model card for more detailed information about the pre-training procedure.

Training

We use the concatenation from multiple datasets to fine-tune our model. In total, we have about 215M (question, answer) pairs. Each dataset was sampled given a weighted probability, and the configuration is detailed in the data_config.json file.

The model was trained with MultipleNegativesRankingLoss using CLS-pooling, dot-product as the similarity function, and a scale of 1.