HPD TinyBERT F128

🚀 HPD-TinyBERT-F128

This repository offers pre - trained models for semantic retrieval sentence representation compression. The sentence embedding model has only 14M parameters and a model size of just 55MB.

🚀 Quick Start

This repository contains the pre - trained models for our paper Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation. The sentence embedding model contains only 14M parameters and the model size is only 55MB.

✨ Features

We propose Homomorphic Projective Distillation (HPD) to learn compressed sentence embeddings. Our method augments a small Transformer encoder model with learnable projection layers to produce compact representations while mimicking a large pre - trained language model to retain the sentence representation quality.

📚 Documentation

This is a sentence - transformers model: It maps sentences & paragraphs to a 128 dimensional dense vector space and can be used for tasks like clustering or semantic search.

The teacher model is [princeton - nlp/sup - simcse - roberta - large](https://huggingface.co/princeton - nlp/sup - simcse - bert - base - uncased) and the student model is [nreimers/TinyBERT_L - 4_H - 312_v2](https://huggingface.co/nreimers/TinyBERT_L - 4_H - 312_v2).

📦 Installation

Using this model becomes easy when you have sentence - transformers installed:

pip install -U sentence-transformers

💻 Usage Examples

Basic Usage

After installing the package, you can simply load our model

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('Xuandong/HPD-TinyBERT-F128')

Advanced Usage

Then you can use our model for encoding sentences into embeddings

sentences = ['He plays guitar.', 'A street vendor is outside.']
sentence_embeddings = model.encode(sentences)

for sentence, embedding in zip(sentences, sentence_embeddings):
    print("Sentence:", sentence)
    print("Embedding:", embedding)
    print("")

📊 Evaluation Results

We evaluate our model on semantic textual similarity (STS) tasks. The results are:

🔧 Technical Details

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 312, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
  (2): Dense({'in_features': 312, 'out_features': 128, 'bias': True, 'activation_function': 'torch.nn.modules.activation.Tanh'})
)

📄 License

The model in this repository is released under the Apache 2.0 license.

📖 Citation

Please cite our paper if you use HPD in your work:

@article{zhao2022compressing,
  title={Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation},
  author={Zhao, Xuandong and Yu, Zhiguo and Wu, Ming and Li, Lei},
  journal={arXiv preprint arXiv:2203.07687},
  year={2022}
}