Ru Longformer Tiny 16384

🚀 Russian Tiny Longformer Model

A compact Longformer model tailored for the Russian language, with extended context support and English comprehension.

🚀 Quick Start

This is a tiny Longformer model crafted specifically for the Russian language. It was initialized using the weights from cointegrated/rubert-tiny2 and has been adjusted to handle a context length of up to 16384 tokens.

We fine - tuned the model on a dataset composed of Russian books, news, Wikipedia articles, and Habr content. Despite being trained on Russian data, it can still understand English, thanks to the source model. For in - depth details, refer to our post on Habr.

✨ Features

• Extended Context: Supports a context length of up to 16384 tokens.
• Multilingual Understanding: Understands both Russian and English.
• Flexible Usage: Can be used as - is for text embeddings or fine - tuned for specific downstream tasks.

📦 Installation

To use this model, you need to install the transformers and sentencepiece libraries. You can install them using the following command:

pip install transformers sentencepiece

💻 Usage Examples

Basic Usage

# pip install transformers sentencepiece
import torch
from transformers import LongformerModel, LongformerTokenizerFast

model = LongformerModel.from_pretrained('kazzand/ru-longformer-tiny-16384')
tokenizer = LongformerTokenizerFast.from_pretrained('kazzand/ru-longformer-tiny-16384')

def get_cls_embedding(text, model, tokenizer, device='cuda'):
    model.to(device)
    batch = tokenizer(text, return_tensors='pt')

    #set global attention for cls token
    global_attention_mask = [
            [1 if token_id == tokenizer.cls_token_id else 0 for token_id in input_ids]
            for input_ids in batch["input_ids"]
        ]

    #add global attention mask to batch
    batch["global_attention_mask"] = torch.tensor(global_attention_mask)

    with torch.no_grad():
        output = model(**batch.to(device))
    return output.last_hidden_state[:,0,:]

🔧 Technical Details

Model Attributes

The model can be directly used to generate text embeddings or further fine - tuned for specific downstream tasks.

📄 Acknowledgments

P.S. Thanks for moral and technical support AbstractDL