Tavbert He

🚀 TavBERT Base Model

A Hebrew BERT - style masked language model that operates over characters. It is pre - trained by masking character spans, similar to SpanBERT (Joshi et al., 2020).

🚀 Quick Start

✨ Features

• It is a Hebrew BERT - style masked language model.
• Operates over characters and is pre - trained by masking character spans.

📦 Installation

The installation steps are not provided in the original README. So, this section is skipped.

💻 Usage Examples

Basic Usage

import numpy as np
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

model = AutoModelForMaskedLM.from_pretrained("tau/tavbert-he")
tokenizer = AutoTokenizer.from_pretrained("tau/tavbert-he")

def mask_sentence(sent, span_len=5):
    start_pos = np.random.randint(0, len(sent) - span_len)
    masked_sent = sent[:start_pos] + '[MASK]' * span_len + sent[start_pos + span_len:]
    print("Masked sentence:", masked_sent)
    output = model(**tokenizer.encode_plus(masked_sent, 
                                           return_tensors='pt'))['logits'][0][1:-1]
    preds = [int(x) for x in torch.argmax(torch.softmax(output, axis=1), axis=1)[start_pos:start_pos + span_len]]
    pred_sent = sent[:start_pos] + ''.join(tokenizer.convert_ids_to_tokens(preds)) + sent[start_pos + span_len:]
    print("Model's prediction:", pred_sent)

📚 Documentation

Training data

The model is trained on the Hebrew section of OSCAR (Ortiz, 2019), which contains 10 GB of text and 20 million sentences.