Legal Hebert Ft

🚀 Legal-HeBERT

Legal-HeBERT is a BERT model tailored for the Hebrew legal and legislative domains. It aims to enhance legal NLP research and facilitate the development of Hebrew legal tools. We've released two versions of Legal-HeBERT. The first one is a fine-tuned variant of HeBERT applied to legal and legislative documents. The second version builds a BERT model from scratch, following the architectural guidelines of HeBERT.

We're continuously collecting legal data, exploring different architectural designs, and creating tagged datasets and legal tasks. These efforts are aimed at evaluating and advancing Hebrew legal tools.

✨ Features

• Two versions of the model are available: a fine - tuned version of HeBERT and a model trained from scratch.
• Intended to boost legal NLP research and tool development in Hebrew.

📦 Installation

No specific installation steps are provided in the original document.

📚 Documentation

📦 Training Data

Our training datasets are as follows:

We express our gratitude to Yair Gardin for referring to the governance data, Elhanan Schwarts for collecting and parsing The Israeli law book, and Jonathan Schler for collecting the judgments of the supreme court.

🔧 Technical Details

Training process

• Vocabulary size: 50,000 tokens
• 4 epochs (1M steps±)
• lr = 5e - 5
• mlm_probability = 0.15
• batch size = 32 (for each gpu)
• NVIDIA GeForce RTX 2080 TI + NVIDIA GeForce RTX 3090 (1 week training)

Additional training settings:

• Fine - tuned HeBERT model: The first eight layers were freezed (like Lee et al. (2019) suggest).
• Legal - HeBERT trained from scratch: The training process is similar to HeBERT and inspired by Chalkidis et al. (2020).

💻 Usage Examples

Basic Usage

# !pip install transformers==4.14.1
from transformers import AutoTokenizer, AutoModel

model_name = 'avichr/Legal-heBERT_ft' # for the fine-tuned HeBERT model 
model_name = 'avichr/Legal-heBERT' # for legal HeBERT model trained from scratch

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

from transformers import pipeline
fill_mask = pipeline(
    "fill-mask",
    model=model_name,
)
fill_mask("הקורונה לקחה את [MASK] ולנו לא נשאר דבר.")

📄 License

No license information is provided in the original document.

Stay tuned!

We're still working on our models and datasets. We'll update this page as we make progress. We're open to collaborations.

Citation

If you use this model, please cite us as follows:

Chriqui, Avihay, Yahav, Inbal and Bar - Siman - Tov, Ittai, Legal HeBERT: A BERT - based NLP Model for Hebrew Legal, Judicial and Legislative Texts (June 27, 2022). Available at: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4147127

@article{chriqui2021hebert,
  title={Legal HeBERT: A BERT-based NLP Model for Hebrew Legal, Judicial and Legislative Texts},
  author={Chriqui, Avihay, Yahav, Inbal and Bar-Siman-Tov, Ittai},
  journal={SSRN preprint:4147127},
  year={2022}
}

Contact us

Avichay Chriqui, The Coller AI Lab
Inbal yahav, The Coller AI Lab
Ittai Bar - Siman - Tov, the BIU Innovation Lab for Law, Data - Science and Digital Ethics

Thank you, תודה, شكرا