Sec Bert Shape

🚀 SEC-BERT

SEC-BERT is a family of BERT models designed for the financial domain. It aims to support financial NLP research and FinTech applications, offering specialized models tailored to handle financial text effectively.

🚀 Quick Start

SEC-BERT consists of the following models:

• SEC-BERT-BASE: Has the same architecture as BERT-BASE and is trained on financial documents.
• SEC-BERT-NUM: Similar to SEC-BERT-BASE, but replaces every number token with a [NUM] pseudo - token to handle all numeric expressions uniformly.
• SEC-BERT-SHAPE (this model): Similar to SEC-BERT-BASE, but replaces numbers with pseudo - tokens representing the number’s shape, preventing fragmentation of numeric expressions, e.g., '53.2' becomes '[XX.X]' and '40,200.5' becomes '[XX,XXX.X]'.

✨ Features

SEC-BERT is specifically designed for the financial domain, which provides more accurate and relevant results for financial NLP tasks compared to general - purpose models. It uses different strategies to handle numbers in financial texts, such as replacing numbers with pseudo - tokens, which can improve the model's performance in understanding and processing financial data.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("nlpaueb/sec-bert-shape")
model = AutoModel.from_pretrained("nlpaueb/sec-bert-shape")

Advanced Usage

import re
import spacy
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("nlpaueb/sec-bert-shape")
spacy_tokenizer = spacy.load("en_core_web_sm")

sentence = "Total net sales decreased 2% or $5.4 billion during 2019 compared to 2018."

def sec_bert_shape_preprocess(text):
    tokens = [t.text for t in spacy_tokenizer(sentence)]

    processed_text = []
    for token in tokens:
        if re.fullmatch(r"(\d+[\d,.]*)|([,.]\d+)", token):
            shape = '[' + re.sub(r'\d', 'X', token) + ']'
            if shape in tokenizer.additional_special_tokens:
                processed_text.append(shape)
            else:
                processed_text.append('[NUM]')
        else:
            processed_text.append(token)
            
    return ' '.join(processed_text)
        
tokenized_sentence = tokenizer.tokenize(sec_bert_shape_preprocess(sentence))
print(tokenized_sentence)
"""
['total', 'net', 'sales', 'decreased', '[X]', '%', 'or', '$', '[X.X]', 'billion', 'during', '[XXXX]', 'compared', 'to', '[XXXX]', '.']
"""

📚 Documentation

Pre - training corpus

The model was pre - trained on 260,773 10 - K filings from 1993 - 2019, publicly available at U.S. Securities and Exchange Commission (SEC).

Pre - training details

• A new vocabulary of 30k subwords was created by training a BertWordPieceTokenizer from scratch on the pre - training corpus.
• BERT was trained using the official code provided in Google BERT's GitHub repository.
• The TF checkpoint was converted to the desired format using Hugging Face's Transformers conversion script, enabling users to load the model in two lines of code for both PyTorch and TF2.
• A model similar to the English BERT - BASE model (12 - layer, 768 - hidden, 12 - heads, 110M parameters) was released.
• The same training set - up was followed: 1 million training steps with batches of 256 sequences of length 512 and an initial learning rate of 1e - 4.
• A single Google Cloud TPU v3 - 8 provided for free from TensorFlow Research Cloud (TRC) was used, along with GCP research credits.

Using SEC - BERT variants as Language Models

🔧 Technical Details

The model's pre - training on a large corpus of 10 - K filings from 1993 - 2019 provides it with a rich understanding of financial language. The creation of a new vocabulary and the use of specific strategies to handle numbers in financial texts contribute to its performance in financial NLP tasks.

📄 License

This model is licensed under the [CC - BY - SA - 4.0](https://creativecommons.org/licenses/by - sa/4.0/) license.