rubert-tiny Open-Source Language Model - Ultra-Small Size, Focused on Russian-English Processing, Fast and Easy to Deploy

Rubert Tiny

Developed by cointegrated

An extremely compact distilled version (45MB, 12M parameters) of the bert-base-multilingual-cased model for Russian and English, prioritizing speed and size over absolute accuracy

Large Language Model

Transformers

Supports Multiple LanguagesOpen Source License:MIT #Russian-English Bilingual Model #Miniature Distilled BERT #Fast Text Embedding

Downloads 36.18k

Release Time : 3/2/2022

Model Overview

This is a distilled miniature BERT model suitable for Russian and English tasks. The model is small, fast, and ideal for simple Russian tasks such as named entity recognition or sentiment classification. Its CLS embedding vectors can be used as sentence representations for Russian-English alignment.

Model Features

Compact Design

Approximately one-tenth the size and speed of standard BERT models, only 45MB in size

Bilingual Support

Supports both Russian and English processing, with embedding vectors enabling bilingual alignment

Multi-task Applicability

Suitable for various downstream tasks, including classification and named entity recognition

Efficient Distillation

Trained using MLM loss, translation ranking loss, and embedding vector distillation techniques

Model Capabilities

Fill-mask

Feature Extraction

Sentence Similarity Calculation

Text Classification

Named Entity Recognition

Use Cases

Natural Language Processing

Russian Text Classification

Performing sentiment analysis or topic classification on Russian texts

Bilingual Sentence Alignment

Using CLS embedding vectors for Russian-English sentence similarity calculation

Named Entity Recognition

Identifying entities such as person names and locations in Russian texts

🚀 Tiny Russian-English Distilled BERT Model

A very small distilled version of the bert-base-multilingual-cased model for Russian and English, suitable for simple tasks with a focus on speed and size.

🚀 Quick Start

This is a very small distilled version of the bert-base-multilingual-cased model for Russian and English (45 MB, 12M parameters). There's also an updated version of this model, rubert-tiny2, which has a larger vocabulary and better performance on almost all Russian NLU tasks.

This model is useful when you want to fine - tune it for a relatively simple Russian task (e.g., NER or sentiment classification), and you prioritize speed and size over accuracy. It's approximately x10 smaller and faster than a base - sized BERT. Its [CLS] embeddings can be used as a sentence representation aligned between Russian and English.

✨ Features

Small and Fast: About x10 smaller and faster than a base - sized BERT.
Cross - language Alignment: The [CLS] embeddings can represent sentences in both Russian and English.
Updated Version Available: The rubert-tiny2 offers better performance on Russian NLU tasks.

🔧 Technical Details

It was trained on the Yandex Translate corpus, OPUS-100 and Tatoeba, using MLM loss (distilled from bert-base-multilingual-cased), translation ranking loss, and [CLS] embeddings distilled from LaBSE, rubert-base-cased-sentence, Laser and USE.

There is a more detailed description in Russian.

💻 Usage Examples

Basic Usage

# pip install transformers sentencepiece
import torch
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("cointegrated/rubert-tiny")
model = AutoModel.from_pretrained("cointegrated/rubert-tiny")
# model.cuda()  # uncomment it if you have a GPU

def embed_bert_cls(text, model, tokenizer):
    t = tokenizer(text, padding=True, truncation=True, return_tensors='pt')
    with torch.no_grad():
        model_output = model(**{k: v.to(model.device) for k, v in t.items()})
    embeddings = model_output.last_hidden_state[:, 0, :]
    embeddings = torch.nn.functional.normalize(embeddings)
    return embeddings[0].cpu().numpy()

print(embed_bert_cls('привет мир', model, tokenizer).shape)
# (312,)

📄 License

This project is licensed under the MIT license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご