roberta-base Open-source Bidirectional Encoder Model - for Russian Feature Extraction, Trained on Large-scale Corpora

Roberta Base

Developed by deepvk

A bidirectional encoder model pretrained specifically for Russian language, trained on large-scale text corpora, supporting feature extraction tasks.

Large Language Model

Transformers

Supports Multiple LanguagesOpen Source License:Apache-2.0 #Russian NLP #Social Text Pretraining #Multisource Corpus

Downloads 44

Release Time : 2/7/2023

Model Overview

This is a Russian pretrained model based on the RoBERTa architecture, primarily used for text feature extraction. The model was trained on a large-scale Russian corpus containing various text types such as social data, Wikipedia, and news articles.

Model Features

Russian Optimization

Specifically pretrained for Russian text, incorporating rich Russian linguistic features

Large-scale Training Data

Trained on 500GB of Russian text data, covering multiple text types and domains

High-performance Architecture

Based on RoBERTa architecture with 12 encoder layers and 768-dimensional embedding space

Model Capabilities

Russian text feature extraction

Contextual semantic understanding

Multidomain text processing

Use Cases

Natural Language Processing

Text Classification

Can be used for Russian text classification tasks

Performs well on the Russian Super Glue benchmark

Semantic Analysis

Suitable for semantic understanding and analysis of Russian text

🚀 RoBERTa-base

A pretrained bidirectional encoder for the Russian language, trained on large text corpora including open social data using the standard MLM objective.

⚠️ Important Note

This model contains only the encoder part without any pretrained head.

Property	Details
Developed by	deepvk
Model Type	RoBERTa
Languages	Mostly Russian and a small fraction of other languages
License	Apache 2.0

🚀 Quick Start

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("deepvk/roberta-base")
model = AutoModel.from_pretrained("deepvk/roberta-base")

text = "Привет, мир!"

inputs = tokenizer(text, return_tensors='pt')
predictions = model(**inputs)

📚 Documentation

Training Details

Training Data

The model was trained on a total of 500 GB of raw text, which is a mix of the following data sources: Wikipedia, Books, Twitter comments, Pikabu, Proza.ru, Film subtitles, News websites, and Social corpus.

Training Hyperparameters

Argument	Value
Training regime	fp16 mixed precision
Training framework	Fairseq
Optimizer	Adam
Adam betas	0.9,0.98
Adam eps	1e-6
Num training steps	500k

The model was trained on a machine with 8xA100 for approximately 22 days.

Architecture details

Argument	Value
Encoder layers	12
Encoder attention heads	12
Encoder embed dim	768
Encoder ffn embed dim	3,072
Activation function	GeLU
Attention dropout	0.1
Dropout	0.1
Max positions	512
Vocab size	50266
Tokenizer type	Byte-level BPE

Evaluation

We evaluated the model on the Russian Super Glue dev set. The best result in each task is marked in bold. All models have the same size except the distilled version of DeBERTa.

Model	RCB	PARus	MuSeRC	TERRa	RUSSE	RWSD	DaNetQA	Score
vk-deberta-distill	0.433	0.56	0.625	0.59	0.943	0.569	0.726	0.635
vk-roberta-base	0.46	0.56	0.679	0.769	0.960	0.569	0.658	0.665
vk-deberta-base	0.450	0.61	0.722	0.704	0.948	0.578	0.76	0.682
vk-bert-base	0.467	0.57	0.587	0.704	0.953	0.583	0.737	0.657
sber-bert-base	0.491	0.61	0.663	0.769	0.962	0.574	0.678	0.678

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご