Bert L12 H256 A4

🚀 Eli4s/Bert-L12-h256-A4

This model is a masked language model pretrained on the bookcorpus dataset using knowledge distillation, with a hidden size of 256 and 4 attention heads.

🚀 Quick Start

This model was pretrained on the bookcorpus dataset using knowledge distillation. It shares the same architecture as BERT but has a hidden size of 256. With 4 attention heads, the head size is 64, similar to the BERT base model. The knowledge distillation was performed using multiple loss functions, and the model's weights were initialized from scratch. The tokenizer is the same as that of the bert-base-uncased model.

💻 Usage Examples

Basic Usage

To load the model & tokenizer:

from transformers import AutoModelForMaskedLM, BertTokenizer

model_name = "eli4s/Bert-L12-h256-A4"
model = AutoModelForMaskedLM.from_pretrained(model_name)
tokenizer = BertTokenizer.from_pretrained(model_name)

Advanced Usage

Use as a Masked Language Model

import torch

sentence = "Let's have a [MASK]."

model.eval()
inputs = tokenizer([sentence], padding='longest', return_tensors='pt')
output = model(inputs['input_ids'], attention_mask=inputs['attention_mask'])

mask_index = inputs['input_ids'].tolist()[0].index(103)
masked_token = output['logits'][0][mask_index].argmax(axis=-1)
predicted_token = tokenizer.decode(masked_token)

print(predicted_token)

Predict the N Most Relevant Predictions

top_n = 5

vocab_size = model.config.vocab_size
logits = output['logits'][0][mask_index].tolist()
top_tokens = sorted(list(range(vocab_size)), key=lambda  i:logits[i], reverse=True)[:top_n]

tokenizer.decode(top_tokens)