%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
EN
Sec Bert Base
SEC-BERT is a series of BERT models tailored for the financial domain, trained on 10-K annual reports from the U.S. Securities and Exchange Commission (SEC), supporting financial natural language processing research and fintech applications.
Downloads 749
Release Time : 3/2/2022
Model Overview
SEC-BERT is a BERT model specifically optimized for the financial domain, primarily used for natural language understanding tasks in financial documents, such as financial report analysis and financial entity recognition.
Model Features
Financial Domain Optimization
Trained specifically on 260,773 10-K annual reports, providing better understanding of financial terminology and expressions.
Multiple Variant Models
Offers three variants: base model (SEC-BERT), numeric processing model (SEC-BERT-NUM), and numeric shape model (SEC-BERT-SHAPE).
Financial Entity Recognition
Excels in financial numeric entity recognition tasks, such as percentages, monetary amounts, and years.
Model Capabilities
Financial Text Understanding
Financial Report Analysis
Financial Entity Recognition
Financial Numeric Processing
Use Cases
Financial Report Analysis
Sales Trend Analysis
Analyzing sales trend changes in annual reports.
Accurately predicts financial trend words such as 'growth' or 'decline'.
Financial Entity Recognition
Financial Numeric Identification
Identifying monetary amounts, percentages, and other numeric information in reports.
Higher accuracy in recognizing numbers and units compared to general BERT models.
๐ SEC-BERT
SEC-BERT is a family of BERT models tailored for the financial domain. It aims to support financial NLP research and FinTech applications, offering specialized models for more accurate financial text analysis.
๐ Quick Start
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("nlpaueb/sec-bert-base")
model = AutoModel.from_pretrained("nlpaueb/sec-bert-base")
โจ Features
SEC-BERT consists of the following models:
- SEC-BERT-BASE (this model): It has the same architecture as BERT-BASE and is trained on financial documents.
- SEC-BERT-NUM: Similar to SEC-BERT-BASE, but it replaces every number token with a [NUM] pseudo - token, handling all numeric expressions uniformly and preventing fragmentation.
- SEC-BERT-SHAPE: Also similar to SEC-BERT-BASE, but it replaces numbers with pseudo - tokens representing the numberโs shape, so numeric expressions (of known shapes) are no longer fragmented. For example, '53.2' becomes '[XX.X]' and '40,200.5' becomes '[XX,XXX.X]'.
๐ฆ Installation
No specific installation steps are provided in the original document.
๐ป Usage Examples
Basic Usage
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("nlpaueb/sec-bert-base")
model = AutoModel.from_pretrained("nlpaueb/sec-bert-base")
Advanced Usage
The following tables show how different SEC - BERT variants perform as language models in various financial text scenarios:
| Sample | Masked Token |
|---|---|
| Total net sales [MASK] 2% or $5.4 billion during 2019 compared to 2018. | decreased |
| Model | Predictions (Probability) |
|---|---|
| BERT-BASE-UNCASED | increased (0.221), were (0.131), are (0.103), rose (0.075), of (0.058) |
| SEC-BERT-BASE | increased (0.678), decreased (0.282), declined (0.017), grew (0.016), rose (0.004) |
| SEC-BERT-NUM | increased (0.753), decreased (0.211), grew (0.019), declined (0.010), rose (0.006) |
| SEC-BERT-SHAPE | increased (0.747), decreased (0.214), grew (0.021), declined (0.013), rose (0.002) |
| Sample | Masked Token |
|---|---|
| Total net sales decreased 2% or $5.4 [MASK] during 2019 compared to 2018. | billion |
| Model | Predictions (Probability) |
|---|---|
| BERT-BASE-UNCASED | billion (0.841), million (0.097), trillion (0.028), ##m (0.015), ##bn (0.006) |
| SEC-BERT-BASE | million (0.972), billion (0.028), millions (0.000), ##million (0.000), m (0.000) |
| SEC-BERT-NUM | million (0.974), billion (0.012), , (0.010), thousand (0.003), m (0.000) |
| SEC-BERT-SHAPE | million (0.978), billion (0.021), % (0.000), , (0.000), millions (0.000) |
| Sample | Masked Token |
|---|---|
| Total net sales decreased [MASK]% or $5.4 billion during 2019 compared to 2018. | 2 |
| Model | Predictions (Probability) |
|---|---|
| BERT-BASE-UNCASED | 20 (0.031), 10 (0.030), 6 (0.029), 4 (0.027), 30 (0.027) |
| SEC-BERT-BASE | 13 (0.045), 12 (0.040), 11 (0.040), 14 (0.035), 10 (0.035) |
| SEC-BERT-NUM | [NUM] (1.000), one (0.000), five (0.000), three (0.000), seven (0.000) |
| SEC-BERT-SHAPE | [XX] (0.316), [XX.X] (0.253), [X.X] (0.237), [X] (0.188), [X.XX] (0.002) |
| Sample | Masked Token |
|---|---|
| Total net sales decreased 2[MASK] or $5.4 billion during 2019 compared to 2018. | % |
| Model | Predictions (Probability) |
|---|---|
| BERT-BASE-UNCASED | % (0.795), percent (0.174), ##fold (0.009), billion (0.004), times (0.004) |
| SEC-BERT-BASE | % (0.924), percent (0.076), points (0.000), , (0.000), times (0.000) |
| SEC-BERT-NUM | % (0.882), percent (0.118), million (0.000), units (0.000), bps (0.000) |
| SEC-BERT-SHAPE | % (0.961), percent (0.039), bps (0.000), , (0.000), bcf (0.000) |
| Sample | Masked Token |
|---|---|
| Total net sales decreased 2% or $[MASK] billion during 2019 compared to 2018. | 5.4 |
| Model | Predictions (Probability) |
|---|---|
| BERT-BASE-UNCASED | 1 (0.074), 4 (0.045), 3 (0.044), 2 (0.037), 5 (0.034) |
| SEC-BERT-BASE | 1 (0.218), 2 (0.136), 3 (0.078), 4 (0.066), 5 (0.048) |
| SEC-BERT-NUM | [NUM] (1.000), l (0.000), 1 (0.000), - (0.000), 30 (0.000) |
| SEC-BERT-SHAPE | [X.X] (0.787), [X.XX] (0.095), [XX.X] (0.049), [X.XXX] (0.046), [X] (0.013) |
| Sample | Masked Token |
|---|---|
| Total net sales decreased 2% or $5.4 billion during [MASK] compared to 2018. | 2019 |
| Model | Predictions (Probability) |
|---|---|
| BERT-BASE-UNCASED | 2017 (0.485), 2018 (0.169), 2016 (0.164), 2015 (0.070), 2014 (0.022) |
| SEC-BERT-BASE | 2019 (0.990), 2017 (0.007), 2018 (0.003), 2020 (0.000), 2015 (0.000) |
| SEC-BERT-NUM | [NUM] (1.000), as (0.000), fiscal (0.000), year (0.000), when (0.000) |
| SEC-BERT-SHAPE | [XXXX] (1.000), as (0.000), year (0.000), periods (0.000), , (0.000) |
| Sample | Masked Token |
|---|---|
| Total net sales decreased 2% or $5.4 billion during 2019 compared to [MASK]. | 2018 |
| Model | Predictions (Probability) |
|---|---|
| BERT-BASE-UNCASED | 2017 (0.100), 2016 (0.097), above (0.054), inflation (0.050), previously (0.037) |
| SEC-BERT-BASE | 2018 (0.999), 2019 (0.000), 2017 (0.000), 2016 (0.000), 2014 (0.000) |
| SEC-BERT-NUM | [NUM] (1.000), year (0.000), last (0.000), sales (0.000), fiscal (0.000) |
| SEC-BERT-SHAPE | [XXXX] (1.000), year (0.000), sales (0.000), prior (0.000), years (0.000) |
| Sample | Masked Token |
|---|---|
| During 2019, the Company [MASK] $67.1 billion of its common stock and paid dividend equivalents of $14.1 billion. | repurchased |
| Model | Predictions (Probability) |
|---|---|
| BERT-BASE-UNCASED | held (0.229), sold (0.192), acquired (0.172), owned (0.052), traded (0.033) |
| SEC-BERT-BASE | repurchased (0.913), issued (0.036), purchased (0.029), redeemed (0.010), sold (0.003) |
| SEC-BERT-NUM | repurchased (0.917), purchased (0.054), reacquired (0.013), issued (0.005), acquired (0.003) |
| SEC-BERT-SHAPE | repurchased (0.902), purchased (0.068), issued (0.010), reacquired (0.008), redeemed (0.006) |
| Sample | Masked Token |
|---|---|
| During 2019, the Company repurchased $67.1 billion of its common [MASK] and paid dividend equivalents of $14.1 billion. | stock |
| Model | Predictions (Probability) |
|---|---|
| BERT-BASE-UNCASED | stock (0.835), assets (0.039), equity (0.025), debt (0.021), bonds (0.017) |
| SEC-BERT-BASE | stock (0.857), shares (0.135), equity (0.004), units (0.002), securities (0.000) |
| SEC-BERT-NUM | stock (0.842), shares (0.157), equity (0.000), securities (0.000), units (0.000) |
| SEC-BERT-SHAPE | stock (0.888), shares (0.109), equity (0.001), securities (0.001), stocks (0.000) |
| Sample | Masked Token |
|---|---|
| During 2019, the Company repurchased $67.1 billion of its common stock and paid [MASK] equivalents of $14.1 billion. | dividend |
| Model | Predictions (Probability) |
|---|---|
| BERT-BASE-UNCASED | cash (0.276), net (0.128), annual (0.083), the (0.040), debt (0.027) |
| SEC-BERT-BASE | dividend (0.890), cash (0.018), dividends (0.016), share (0.013), tax (0.010) |
| SEC-BERT-NUM | dividend (0.735), cash (0.115), share (0.087), tax (0.025), stock (0.013) |
| SEC-BERT-SHAPE | dividend (0.655), cash (0.248), dividends (0.042), share (0.019), out (0.003) |
| Sample | Masked Token |
|---|---|
| During 2019, the Company repurchased $67.1 billion of its common stock and paid dividend [MASK] of $14.1 billion. | equivalents |
| Model | Predictions (Probability) |
|---|---|
| BERT-BASE-UNCASED | revenue (0.085), earnings (0.078), rates (0.065), amounts (0.064), proceeds (0.062) |
| SEC-BERT-BASE | payments (0.790), distributions (0.087), equivalents (0.068), cash (0.013), amounts (0.004) |
| SEC-BERT-NUM | payments (0.845), equivalents (0.097), distributions (0.024), increases (0.005), dividends (0.004) |
| SEC-BERT-SHAPE | payments (0.784), equivalents (0.093), distributions (0.043), dividends (0.015), requirements (0.009) |
๐ Documentation
Pre - training corpus
The model was pre - trained on 260,773 10 - K filings from 1993 - 2019, publicly available at U.S. Securities and Exchange Commission (SEC)
Pre - training details
- A new vocabulary of 30k subwords was created by training a BertWordPieceTokenizer from scratch on the pre - training corpus.
- BERT was trained using the official code provided in Google BERT's GitHub repository.
- The Hugging Face's Transformers conversion script was used to convert the TF checkpoint into the desired format, enabling users of both PyTorch and TF2 to load the model in two lines of code.
- A model similar to the English BERT - BASE model (12 - layer, 768 - hidden, 12 - heads, 110M parameters) was released.
- The same training set - up was followed: 1 million training steps with batches of 256 sequences of length 512 and an initial learning rate of 1e - 4.
- A single Google Cloud TPU v3 - 8 provided for free from [TensorFlow Research Cloud (TRC)]((https://sites.research.google/trc) was used, along with GCP research credits.
๐ง Technical Details
The technical details involve the pre - training process, including vocabulary creation, model training using official code, checkpoint conversion, and the specific training setup with the use of Google Cloud TPU and research credits.
๐ License
This project is licensed under the cc - by - sa - 4.0 license.
๐ Publication
If you use this model, please cite the following article: FiNER: Financial Numeric Entity Recognition for XBRL Tagging Lefteris Loukas, Manos Fergadiotis, Ilias Chalkidis, Eirini Spyropoulou, Prodromos Malakasiotis, Ion Androutsopoulos and George Paliouras In the Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022) (Long Papers), Dublin, Republic of Ireland, May 22 - 27, 2022
@inproceedings{loukas-etal-2022-finer,
Phi 2 GGUF
Other
Phi-2 is a small yet powerful language model developed by Microsoft, featuring 2.7 billion parameters, focusing on efficient inference and high-quality text generation.
Large Language Model Supports Multiple Languages
P
TheBloke
41.5M
205
Roberta Large
MIT
A large English language model pre-trained with masked language modeling objectives, using improved BERT training methods
Large Language Model English
R
FacebookAI
19.4M
212
Distilbert Base Uncased
Apache-2.0
DistilBERT is a distilled version of the BERT base model, maintaining similar performance while being more lightweight and efficient, suitable for natural language processing tasks such as sequence classification and token classification.
Large Language Model English
D
distilbert
11.1M
669
Llama 3.1 8B Instruct GGUF
Meta Llama 3.1 8B Instruct is a multilingual large language model optimized for multilingual dialogue use cases, excelling in common industry benchmarks.
Large Language Model English
L
modularai
9.7M
4
Xlm Roberta Base
MIT
XLM-RoBERTa is a multilingual model pretrained on 2.5TB of filtered CommonCrawl data across 100 languages, using masked language modeling as the training objective.
Large Language Model Supports Multiple Languages
X
FacebookAI
9.6M
664
Roberta Base
MIT
An English pre-trained model based on Transformer architecture, trained on massive text through masked language modeling objectives, supporting text feature extraction and downstream task fine-tuning
Large Language Model English
R
FacebookAI
9.3M
488
Opt 125m
Other
OPT is an open pre-trained Transformer language model suite released by Meta AI, with parameter sizes ranging from 125 million to 175 billion, designed to match the performance of the GPT-3 series while promoting open research in large-scale language models.
Large Language Model English
O
facebook
6.3M
198
1
A pretrained model based on the transformers library, suitable for various NLP tasks
Large Language Model
Transformers
Transformers1
unslothai
6.2M
1
Llama 3.1 8B Instruct
Llama 3.1 is Meta's multilingual large language model series, featuring 8B, 70B, and 405B parameter scales, supporting 8 languages and code generation, with optimized multilingual dialogue scenarios.
Large Language Model Transformers Supports Multiple Languages
Transformers Supports Multiple LanguagesL
meta-llama
5.7M
3,898
T5 Base
Apache-2.0
The T5 Base Version is a text-to-text Transformer model developed by Google with 220 million parameters, supporting multilingual NLP tasks.
Large Language Model Supports Multiple Languages
T
google-t5
5.4M
702
Featured Recommended AI Models
ยฉ 2025AIbase