Bert L12 H384 A6

🚀 Eli4s/Bert-L12-h384-A6 Model

This model is a knowledge-distilled variant of BERT, offering efficient performance with a reduced hidden size and the same head size as BERT.

🚀 Quick Start

This model was pretrained on the BookCorpus dataset using knowledge distillation. It shares the same architecture as BERT but has a hidden size of 384 (half of BERT's hidden size) and 6 attention heads, resulting in the same head size as BERT. The knowledge distillation process employed multiple loss functions, and the model's weights were initialized from scratch. Note that the tokenizer used is the same as that of the bert-base-uncased model.

💻 Usage Examples

Basic Usage

To load the model and tokenizer:

from transformers import AutoModelForMaskedLM, BertTokenizer

model_name = "eli4s/Bert-L12-h384-A6"
model = AutoModelForMaskedLM.from_pretrained(model_name)
tokenizer = BertTokenizer.from_pretrained(model_name)

To use it on a sentence:

import torch

sentence = "Let's have a [MASK]."

model.eval()
inputs = tokenizer([sentence], padding='longest', return_tensors='pt')
output = model(inputs['input_ids'], attention_mask=inputs['attention_mask'])

mask_index = inputs['input_ids'].tolist()[0].index(103)
masked_token = output['logits'][0][mask_index].argmax(axis=-1)
predicted_token = tokenizer.decode(masked_token)

print(predicted_token)

Advanced Usage

We can also predict the n most relevant predictions:

top_n = 5

vocab_size = model.config.vocab_size
logits = output['logits'][0][mask_index].tolist()
top_tokens = sorted(list(range(vocab_size)), key=lambda  i:logits[i], reverse=True)[:top_n]

tokenizer.decode(top_tokens)