Legal Hebert

🚀 Legal-HeBERT

Legal-HeBERT is a BERT model tailored for the Hebrew legal and legislative domains. It aims to enhance legal NLP research and tool development in Hebrew.

🚀 Quick Start

Legal-HeBERT offers two versions. The first one is a fine - tuned model of HeBERT applied to legal and legislative documents. The second version trains a BERT model from scratch following HeBERT's architecture guidelines. We are continuously collecting legal data, exploring different architectural designs, and conducting tagged datasets and legal tasks for evaluating and developing Hebrew legal tools.

✨ Features

• Domain - Specific: Specifically designed for Hebrew legal and legislative domains.
• Two Versions: A fine - tuned version and a model trained from scratch.

📦 Training Data

Our training datasets are as follows:

We express our gratitude to Yair Gardin for referring to the governance data, Elhanan Schwarts for collecting and parsing The Israeli law book, and Jonathan Schler for collecting the judgments of the supreme court.

🔧 Training Process

• Vocabulary size: 50,000 tokens
• Epochs: 4 epochs (1M steps±)
• Learning rate: lr = 5e - 5
• MLM probability: mlm_probability = 0.15
• Batch size: 32 (for each gpu)
• Hardware: NVIDIA GeForce RTX 2080 TI + NVIDIA GeForce RTX 3090 (1 week training)

Additional training settings:

• Fine - tuned HeBERT model: The first eight layers were freezed (like Lee et al. (2019) suggest).
• Legal - HeBERT trained from scratch: The training process is similar to HeBERT and inspired by Chalkidis et al. (2020).

💻 Usage Examples

Basic Usage

The models can be found in huggingface hub and can be fine - tuned to any down - stream task:

# !pip install transformers==4.14.1
from transformers import AutoTokenizer, AutoModel

model_name = 'avichr/Legal-heBERT_ft' # for the fine-tuned HeBERT model 
model_name = 'avichr/Legal-heBERT' # for legal HeBERT model trained from scratch

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

from transformers import pipeline
fill_mask = pipeline(
    "fill-mask",
    model=model_name,
)
fill_mask("הקורונה לקחה את [MASK] ולנו לא נשאר דבר.")

📚 Stay tuned!

We are still working on our models and the datasets. We will update this page as we make progress. We are open for collaborations.

📄 License

If you used this model, please cite us as follows: Chriqui, Avihay, Yahav, Inbal and Bar - Siman - Tov, Ittai, Legal HeBERT: A BERT - based NLP Model for Hebrew Legal, Judicial and Legislative Texts (June 27, 2022). Available at: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4147127

@article{chriqui2021hebert,
  title={Legal HeBERT: A BERT-based NLP Model for Hebrew Legal, Judicial and Legislative Texts},
  author={Chriqui, Avihay, Yahav, Inbal and Bar-Siman-Tov, Ittai},
  journal={SSRN preprint:4147127},
  year={2022}
}

📞 Contact us

• Avichay Chriqui, The Coller AI Lab
• Inbal yahav, The Coller AI Lab
• [Ittai Bar - Siman - Tov](mailto:Ittai.Bar - Siman - Tov@biu.ac.il), the BIU Innovation Lab for Law, Data - Science and Digital Ethics

Thank you, תודה, شكرا