B

Bert 1.3b

Developed by retrieva-jp
Transformer encoder pretrained based on Megatron-LM, specifically designed for Japanese scenarios
Downloads 56
Release Time : 6/25/2024

Model Overview

RetrievaBERT is a Transformer encoder pretrained on the Megatron-LM framework, primarily targeting Japanese application scenarios, featuring advanced characteristics such as pre-normalization and SwiGLU activation functions

Model Features

Pre-normalization (PreNorm)
Improves training stability
SwiGLU activation function
Enhances model expressiveness
Grouped query attention mechanism
Efficient attention computation
Long text processing capability
Supports long text processing up to 2048 tokens

Model Capabilities

Japanese text understanding
English text understanding
Masked language modeling
Downstream task fine-tuning

Use Cases

Text understanding
Japanese text classification
Can be used for tasks such as Japanese sentiment analysis and topic classification
Achieved 0.959 accuracy on the MARC-ja task
Semantic similarity calculation
Can be used to calculate semantic similarity between Japanese text pairs
Pearson correlation coefficient of 0.917 on the JSTS task
Question answering systems
Japanese question answering system
Can be used to build Japanese-based question answering systems
EM score of 0.875 on the JSQuAD task
Featured Recommended AI Models
ยฉ 2025AIbase