Bert Base Uncased Echr

🚀 LEGAL-BERT: The Muppets straight out of Law School

LEGAL-BERT is a family of BERT models designed for the legal domain, aiming to support legal NLP research, computational law, and legal technology applications.

🚀 Quick Start

To load the pre-trained LEGAL-BERT model, you can use the following code:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("nlpaueb/bert-base-uncased-echr")
model = AutoModel.from_pretrained("nlpaueb/bert-base-uncased-echr")

✨ Features

• LEGAL-BERT is a collection of BERT models tailored for the legal domain, assisting in legal NLP research, computational law, and legal technology applications.
• Different sub - domain variants (CONTRACTS-, EURLEX-, ECHR-) and the general LEGAL-BERT perform better than vanilla BERT for domain - specific tasks.
• This specific variant is pre - trained on ECHR cases.

📚 Documentation

Pre - training corpora

The pre - training corpora of LEGAL-BERT include:

• 116,062 documents of EU legislation, publicly available from EURLEX (http://eur-lex.europa.eu), the repository of EU Law under the EU Publication Office.
• 61,826 documents of UK legislation, publicly available from the UK legislation portal (http://www.legislation.gov.uk).
• 19,867 cases from the European Court of Justice (ECJ), also available from EURLEX.
• 12,554 cases from HUDOC, the repository of the European Court of Human Rights (ECHR) (http://hudoc.echr.coe.int/eng).
• 164,141 cases from various courts across the USA, hosted in the Case Law Access Project portal (https://case.law).
• 76,366 US contracts from EDGAR, the database of US Securities and Exchange Commission (SECOM) (https://www.sec.gov/edgar.shtml).

Pre - training details

• We trained BERT using the official code from Google BERT's GitHub repository (https://github.com/google-research/bert).
• We released a model similar to the English BERT - BASE model (12 - layer, 768 - hidden, 12 - heads, 110M parameters).
• We followed the same training setup: 1 million training steps with batches of 256 sequences of length 512 and an initial learning rate of 1e - 4.
• We used a single Google Cloud TPU v3 - 8 provided for free from TensorFlow Research Cloud (TFRC) and GCP research credits.

Models list

* LEGAL - BERT - BASE is the model referred to as LEGAL - BERT - SC in Chalkidis et al. (2020); a model trained from scratch in the legal corpora mentioned below using a newly created vocabulary by a sentence - piece tokenizer trained on the very same corpora.

** The LEGAL - BERT - FP models have been released in Archive.org (https://archive.org/details/legal_bert_fp).

Use LEGAL - BERT variants as Language Models

Evaluation on downstream tasks

Refer to the experiments in the article "LEGAL - BERT: The Muppets straight out of Law School" by Chalkidis et al., 2020 (https://aclanthology.org/2020.findings - emnlp.261).

Author - Publication

@inproceedings{chalkidis - etal - 2020 - legal,
    title = "{LEGAL}-{BERT}: The Muppets straight out of Law School",
    author = "Chalkidis, Ilias  and
      Fergadiotis, Manos  and
      Malakasiotis, Prodromos  and
      Aletras, Nikolaos  and
      Androutsopoulos, Ion",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2020",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    doi = "10.18653/v1/2020.findings - emnlp.261",
    pages = "2898--2904"
}

About Us

AUEB's Natural Language Processing Group develops algorithms, models, and systems for natural language processing and generation.

The group's research interests include:

• Question answering systems for various data sources, especially biomedical question answering.
• Natural language generation from databases and ontologies, especially Semantic Web ontologies.
• Text classification, including spam and abusive content filtering.
• Information extraction and opinion mining, including legal text analytics and sentiment analysis.
• Natural language processing tools for Greek, such as parsers and named - entity recognizers.
• Machine learning in natural language processing, especially deep learning.

The group is part of the Information Processing Laboratory of the Department of Informatics of the Athens University of Economics and Business.

Ilias Chalkidis on behalf of AUEB's Natural Language Processing Group

| Github: @ilias.chalkidis | Twitter: @KiddoThe2B |

📄 License

This project is licensed under the CC - BY - SA - 4.0 license.