AIbase
EN
Home
Dense Encoder Distilbert Frozen Emb
D

Dense Encoder Distilbert Frozen Emb

Developed by vocab-transformers
Dense retrieval model based on DistilBERT architecture, trained on the MS MARCO dataset with frozen word embedding layers
Text Embedding
Transformers
#Information Retrieval Optimization#Frozen Word Embeddings#MarginMSE Loss
Downloads 26
Release Time : 4/5/2022

Model Overview

This model is a variant of DistilBERT, specifically optimized for information retrieval tasks, trained using the MarginMSE loss function, suitable for generating dense vector representations of documents and queries

Model Features

Frozen Word Embeddings Training
Keeps pre-trained word embedding layer parameters unchanged during training, potentially improving model stability
MarginMSE Optimization
Trained using the MarginMSE loss function, specifically optimizing ranking performance for retrieval tasks
Lightweight Architecture
Based on the DistilBERT architecture, smaller and faster than the original BERT model while maintaining good performance

Model Capabilities

Text Vector Representation
Semantic Similarity Calculation
Information Retrieval
Document Ranking

Use Cases

Search Engines
Web Search Result Ranking
Generates dense vector representations of queries and documents for search engines to use in relevance ranking
Performs well in standard retrieval evaluations such as TREC-DL
Question Answering Systems
Answer Passage Retrieval
Quickly retrieves passages related to questions from a large corpus of documents
Demonstrates stable performance on financial QA datasets like FiQA
Featured Recommended AI Models
AIbase
Empowering the Future, Your AI Solution Knowledge Base
English简体中文繁體中文にほんご
© 2025AIbase