%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
Msmarco Distilbert Word2vec256k MLM 230k
This model is a pre-trained language model based on the DistilBERT architecture, initialized with a 256k vocabulary using word2vec and trained on the MS MARCO corpus with masked language modeling (MLM).
Downloads 16
Release Time : 3/2/2022
Model Overview
This model combines word2vec embedding initialization with DistilBERT's lightweight architecture, making it suitable for text representation and semantic understanding tasks.
Model Features
Word2Vec Initialization
Initializes a 256k vocabulary using word2vec, improving the quality of word embeddings.
Lightweight Architecture
Based on the DistilBERT architecture, it is more lightweight and efficient compared to the original BERT model.
Large-Scale Training
Trained for 230,000 steps with MLM on the MS MARCO corpus, providing strong semantic understanding capabilities.
Frozen Word Embeddings
Keeps the word embedding matrix frozen during training, focusing on optimizing higher-level structures.
Model Capabilities
Text Representation
Semantic Understanding
Masked Language Modeling
Use Cases
Information Retrieval
Document Retrieval
Can be used to build efficient document retrieval systems by understanding the semantic relationships between queries and documents.
Question Answering
Open-Domain QA
Can serve as a semantic understanding component in QA systems, helping to interpret questions and retrieve relevant answers.
Featured Recommended AI Models