Dense Encoder Msmarco Distilbert Word2vec256k Emb Updated

🚀 dense_encoder-msmarco-distilbert-word2vec256k

This model is based on msmarco-word2vec256000-distilbert-base-uncased with a 256k sized vocabulary initialized with word2vec. It can be used for tasks like sentence similarity, clustering, or semantic search.

🚀 Quick Start

Prerequisites

You need to install sentence-transformers to use this model easily:

pip install -U sentence-transformers

Basic Usage

from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('{MODEL_NAME}')
embeddings = model.encode(sentences)
print(embeddings)

✨ Features

• Based on Specific Model: This model is based on msmarco-word2vec256000-distilbert-base-uncased with a 256k sized vocabulary initialized with word2vec.
• Trained on MS MARCO: It has been trained on MS MARCO using MarginMSELoss.
• High Performance:
  • MS MARCO dev: 34.51 (MRR@10)
  • TREC-DL 2019: 66.12 (nDCG@10)
  • TREC-DL 2020: 68.62 (nDCG@10)

📦 Installation

If you haven't installed sentence-transformers, you can use the following command:

pip install -U sentence-transformers

💻 Usage Examples

Basic Usage (Sentence-Transformers)

from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('{MODEL_NAME}')
embeddings = model.encode(sentences)
print(embeddings)

Advanced Usage (HuggingFace Transformers)

from transformers import AutoTokenizer, AutoModel
import torch


#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)


# Sentences we want sentence embeddings for
sentences = ['This is an example sentence', 'Each sentence is converted']

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
model = AutoModel.from_pretrained('{MODEL_NAME}')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, mean pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)

📚 Documentation

Evaluation Results

For an automated evaluation of this model, see the Sentence Embeddings Benchmark: https://seb.sbert.net

Training

The model was trained with the following parameters:

DataLoader: torch.utils.data.dataloader.DataLoader of length 7858 with parameters:

{'batch_size': 64, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}

Loss: sentence_transformers.losses.MarginMSELoss.MarginMSELoss

Parameters of the fit()-Method:

{
    "epochs": 30,
    "evaluation_steps": 0,
    "evaluator": "NoneType",
    "max_grad_norm": 1,
    "optimizer_class": "<class 'transformers.optimization.AdamW'>",
    "optimizer_params": {
        "lr": 2e-05
    },
    "scheduler": "WarmupLinear",
    "steps_per_epoch": null,
    "warmup_steps": 1000,
    "weight_decay": 0.01
}

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 250, 'do_lower_case': False}) with Transformer model: DistilBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
)

🔧 Technical Details

This model is based on the msmarco-word2vec256000-distilbert-base-uncased model, which has a 256k sized vocabulary initialized with word2vec. It is trained on the MS MARCO dataset using MarginMSELoss. The training process involves specific parameters for the DataLoader, Loss, and the fit() method, which are detailed in the "Training" section. The model architecture consists of a Transformer layer followed by a Pooling layer.

📄 License

No license information provided in the original document.