Bert L12 H240 A12

🚀 Eli4s/Bert-L12-h240-A12 Model

This model is pretrained using knowledge distillation on the bookcorpus dataset, offering a unique architecture with specific hidden size and attention head configurations.

🚀 Quick Start

This model was pretrained on the bookcorpus dataset using knowledge distillation. It shares the same architecture as BERT but has a hidden size of 240. With 12 attention heads, the head size is 20, different from the BERT base model's 64. The knowledge distillation was performed using multiple loss functions, and the model weights were initialized from scratch. The tokenizer is the same as that of the bert-base-uncased model.

📦 Installation

No specific installation steps are provided in the original text. However, you need to have the transformers library installed to use this model. You can install it using the following command:

pip install transformers

💻 Usage Examples

Basic Usage

To load the model and tokenizer:

from transformers import AutoModelForMaskedLM, BertTokenizer

model_name = "eli4s/Bert-L12-h240-A12"
model = AutoModelForMaskedLM.from_pretrained(model_name)
tokenizer = BertTokenizer.from_pretrained(model_name)

Advanced Usage

Use it as a masked language model

import torch

sentence = "Let's have a [MASK]."

model.eval()
inputs = tokenizer([sentence], padding='longest', return_tensors='pt')
output = model(inputs['input_ids'], attention_mask=inputs['attention_mask'])

mask_index = inputs['input_ids'].tolist()[0].index(103)
masked_token = output['logits'][0][mask_index].argmax(axis=-1)
predicted_token = tokenizer.decode(masked_token)

print(predicted_token)

Predict the n most relevant predictions

top_n = 5

vocab_size = model.config.vocab_size
logits = output['logits'][0][mask_index].tolist()
top_tokens = sorted(list(range(vocab_size)), key=lambda  i:logits[i], reverse=True)[:top_n]

tokenizer.decode(top_tokens)

🔧 Technical Details

The model is based on the BERT architecture but has a hidden size of 240 and 12 attention heads with a head size of 20. Knowledge distillation was employed during the pretraining process, using multiple loss functions. The weights of the model were initialized from scratch. The tokenizer is the same as that of the bert-base-uncased model.