Vinilm 2021 From Large

🚀 viniLM-2021-from-large

A VMware-specific pretrained language model distilled from vBERT-2021-large for faster inference.

🚀 Quick Start

The viniLM-2021-from-large model is a VMware-specific Language Model. It is distilled from vBERT-2021-large to achieve faster inference times without significant performance loss.

✨ Features

• Distilled Model: Based on MiniLMv2 distillation, it distills vBERT-2021-large into a smaller minilmv2 model.
• Fast Inference: Enables quicker inference while maintaining good performance.
• VMware-specific: Designed for VMware-related NLP tasks.

📦 Installation

To use this model, you need to have the transformers library installed. You can install it using pip:

pip install transformers

💻 Usage Examples

Basic Usage

Here is how to use this model to get the features of a given text in PyTorch:

from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('VMware/vinilm-2021-from-large')
model = BertModel.from_pretrained("VMware/vinilm-2021-from-large")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

And in TensorFlow:

from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('VMware/vinilm-2021-from-large')
model = TFBertModel.from_pretrained('VMware/vinilm-2021-from-large')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)

📚 Documentation

Model Info

Motivation

Based on MiniLMv2 distillation, we have distilled vBERT-2021-large into a smaller minilmv2 model for faster inference times without a significant loss of performance.

Intended Use

The model functions as a VMware-specific Language Model.

Training

• Distilled From: vBERT-2021-large
• Initial Weights: nreimers/MiniLMv2-L6-H768-distilled-from-BERT-Large

Datasets

Publically available VMware text data such as VMware Docs, Blogs, etc. were used for distilling the teacher vBERT-2021-large model into vinilm-2021-from-large model. Sourced in May 2021. (~320,000 Documents)

Preprocessing

• Decoding HTML
• Decoding Unicode
• Stripping repeated characters
• Splitting compound word
• Spelling correction

Model performance measures

We benchmarked vinilm on various VMware-specific NLP downstream tasks (IR, classification, etc).

🔧 Technical Details

Since the model is distilled from a vBERT model based on the BERT model, it may have the same biases embedded within the original BERT model. The data needs to be preprocessed using our internal vNLP Preprocessor (not available to the public) to maximize its performance.

📄 License

This model is licensed under the Apache 2.0 license.