D

Deberta V1 Base

Developed by deepvk
DeBERTa-base is a pre-trained bidirectional encoder for Russian, mainly used for processing Russian text tasks.
Downloads 160
Release Time : 2/7/2023

Model Overview

This model is trained on a large text corpus containing open social data using the standard Masked Language Model (MLM) objective and supports Russian and a small number of other languages.

Model Features

Large-scale training data
Trained using 400GB of filtered and deduplicated text data from multiple sources, including Wikipedia, books, Twitter comments, etc.
Efficient deduplication process
Data deduplication is performed using MinHash and Jaccard similarity calculations to ensure the diversity of the training data.
High-performance optimization
The AdamW optimizer and mixed-precision training are used, and the model is trained on 8 A100s for 30 days to achieve efficient training results.

Model Capabilities

Russian text processing
Masked Language Model
Text encoding

Use Cases

Natural Language Processing
Russian text classification
Can be used for Russian text classification tasks, such as sentiment analysis and topic classification.
Performs excellently on the Russian Super Glue development set.
Text embedding
Generates embedding representations of Russian text for downstream tasks such as similarity calculation and clustering.
Featured Recommended AI Models
ยฉ 2025AIbase