Vbert 2021 Base

🚀 vBERT-2021-BASE

A VMware-specific pretrained language model addressing issues with traditional BERT models.

🚀 Quick Start

The vBERT-2021-BASE model is a VMware-specific Language Model. It's designed to overcome the limitations of traditional BERT models when dealing with VMware-specific words, technical terms, and compound words.

✨ Features

• Custom Vocabulary: Replaced the first 1k unused tokens of BERT's vocabulary with VMware-specific terms.
• Domain-Specific Pretraining: Pretrained on VMware domain data for 78K steps (approximately 5 epochs).
• Improved Performance: Scored higher than the 'bert-base-uncased' model on various VMware-specific NLP downstream tasks.

📦 Installation

To use this model, you need to install the transformers library. You can install it using pip:

pip install transformers

💻 Usage Examples

Basic Usage

Here is how to use this model to get the features of a given text in PyTorch:

from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('VMware/vbert-2021-base')
model = BertModel.from_pretrained("VMware/vbert-2021-base")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

Advanced Usage

Here is how to use this model in TensorFlow:

from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('VMware/vbert-2021-base')
model = TFBertModel.from_pretrained('VMware/vbert-2021-base')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)

📚 Documentation

Model Info

Motivation

Traditional BERT models struggle with VMware-specific words (Tanzu, vSphere, etc.), technical terms, and compound words. (Weaknesses of WordPiece Tokenization)

We have pretrained our vBERT model to address the aforementioned issues using our BERT Pretraining Library. We replaced the first 1k unused tokens of BERT's vocabulary with VMware-specific terms to create a modified vocabulary. Then we pretrained the 'bert-base-uncased' model for an additional 78K steps (71k With MSL_128 and 7k with MSL_512) (approximately 5 epochs) on VMware domain data.

Intended Use

The model functions as a VMware-specific Language Model.

Training

Datasets

Publically available VMware text data such as VMware Docs, Blogs etc. were used for creating the pretraining corpus. Sourced in May, 2021. (~320,000 Documents)

Preprocessing

• Decoding HTML
• Decoding Unicode
• Stripping repeated characters
• Splitting compound word
• Spelling correction

Model performance measures

We benchmarked vBERT on various VMware-specific NLP downstream tasks (IR, classification, etc). The model scored higher than the 'bert-base-uncased' model on all benchmarks.

Limitations and bias

⚠️ Important Note

Since the model is further pretrained on the BERT model, it may have the same biases embedded within the original BERT model.

💡 Usage Tip

The data needs to be preprocessed using our internal vNLP Preprocessor (not available to the public) to maximize its performance.

📄 License

This project is licensed under the Apache 2.0 License.