Ru Longformer Base 4096

🚀 Russian Longformer Base Model

This is a Longformer base model specifically designed for the Russian language, offering extended context support and fine - tuned on Russian book datasets.

🚀 Quick Start

This is a base Longformer model tailored for the Russian language. It was initialized using the weights from blinoff/roberta-base-russian-v0 and has been adjusted to handle a context length of up to 4096 tokens. We fine - tuned this model on a dataset of Russian books. For more in - depth details, please refer to our post on Habr.

✨ Features

• Extended Context: Supports a context length of up to 4096 tokens.
• Fine - Tuned: Fine - tuned on a dataset of Russian books.
• Versatile Use: Can be used directly for text embeddings or fine - tuned for specific downstream tasks.

📦 Installation

To use this model, you need to install the necessary libraries. You can do this using the following command:

pip install transformers sentencepiece

💻 Usage Examples

Basic Usage

The following code demonstrates how to generate text embeddings using the model:

# pip install transformers sentencepiece
import torch
from transformers import LongformerForMaskedLM, LongformerTokenizerFast

model = LongformerModel.from_pretrained('kazzand/ru-longformer-base-4096')
tokenizer = LongformerTokenizerFast.from_pretrained('kazzand/ru-longformer-base-4096')

def get_cls_embedding(text, model, tokenizer, device='cuda'):
    model.to(device)
    batch = tokenizer(text, return_tensors='pt')

    #set global attention for cls token
    global_attention_mask = [
            [1 if token_id == tokenizer.cls_token_id else 0 for token_id in input_ids]
            for input_ids in batch["input_ids"]
        ]

    #add global attention mask to batch
    batch["global_attention_mask"] = torch.tensor(global_attention_mask)

    with torch.no_grad():
        output = model(**batch.to(device))
    return output.last_hidden_state[:,0,:]

🔧 Technical Details

Model Attributes

The model can be used out - of - the - box to generate text embeddings or can be further fine - tuned for a particular downstream task.