Sbertdistil

🚀 FractalGPT/SbertDistil

This is a sentence-transformers model that maps sentences and paragraphs into a 384-dimensional dense vector space. It can be used for tasks such as clustering or semantic search. It's a fast and compact model designed to solve the problem of determining sentence proximity. In the future, we'll further optimize it for size and speed. Project

🚀 Quick Start

This model simplifies usage once you have sentence-transformers installed:

• Run example in Collab

📦 Installation

pip install -U sentence-transformers

💻 Usage Examples

Basic Usage

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('FractalGPT/SbertDistil')

def cos(x, y):
  return np.dot(x, y)/(np.linalg.norm(x)*np.linalg.norm(y))

text_1 = "Кто такой большой кот?"
text_2 = "Who is kitty?"
a = model.encode(text_1)
b = model.encode(text_2)
cos(a, b)

>>> 0.8072159157330788

🔧 Technical Details

Training

• The original weights were taken from cointegrated/rubert-tiny2.
• Training was conducted in two stages:

1. In the first stage, the model was trained on Wikipedia texts (4 million texts) for three epochs.
2. In the second stage, training was conducted on Wikipedia and a dialog dataset for one epoch.

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 312, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
  (2): Dense({'in_features': 312, 'out_features': 384, 'bias': True, 'activation_function': 'torch.nn.modules.linear.Identity'})
)

📄 License

This project is licensed under the Apache-2.0 license.