BusinessBERT Open-Source Business Language Model - Pre-trained to Integrate Industry Information and Optimize NLP Tasks

Businessbert

Developed by pborchert

An industry-sensitive language model for business applications, pre-trained on business communication corpora, integrating industry information to optimize business-related NLP tasks.

Large Language Model

Transformers

English#Industry-sensitive language model #Business text analysis #Financial NLP

Downloads 1,782

Release Time : 1/12/2024

Model Overview

Business BERT is a language model focused on the business domain, integrating industry information through pre-training, suitable for business-related NLP tasks such as sequence classification, named entity recognition, and sentiment analysis.

Model Features

Industry-sensitive pre-training

Incorporates industry classification (IC) as an additional pre-training objective to embed industry information and enhance performance in business-related tasks.

Large-scale business corpus

Pre-trained on a 2.23 billion-token business communication corpus, including annual disclosure documents, corporate website content, and scientific literature.

Multi-task adaptation

Supports various business-related NLP tasks, including classification, named entity recognition, sentiment analysis, and question-answering systems.

Model Capabilities

Text classification

Named entity recognition

Sentiment analysis

Question-answering system

Industry classification

Use Cases

Financial analysis

Financial risk assessment

Analyze company disclosure documents to assess financial risk

F1 score 85.89, accuracy 87.02

Liquidity ratio analysis

Understand and interpret financial metrics such as liquidity ratios

Market research

News headline topic classification

Classify business news headlines

F1 score 75.06, accuracy 67.71

Industry classification

Identify the industry to which a company belongs

Investment decision

Sentiment analysis

Analyze the sentiment of financial texts

FiQA dataset mean squared error 0.0758, mean absolute error 0.202

Stock tweet analysis

Analyze discussions about stocks on social media

F1 score 69.14, accuracy 69.54

🚀 BusinessBERT

An industry-sensitive language model for business applications pretrained on business communication corpora. It incorporates industry classification as a pretraining objective besides masked language modeling.

🚀 Quick Start

The widget can be used to test the model:

"Sanofi is in the [MASK] industry."
"The current ratio measures [MASK]."

✨ Features

An industry-sensitive language model for business applications.
Incorporates industry classification (IC) as a pretraining objective besides masked language modeling (MLM).

📚 Documentation

Model description

We introduce BusinessBERT, an industry-sensitive language model for business applications. The advantage of the model is the training approach focused on incorporating industry information relevant for business related natural language processing (NLP) tasks. We compile three large-scale textual corpora consisting of annual disclosures, company website content and scientific literature representing business communication. In total, the corpora include 2.23 billion token. BusinessBERT builds upon the bidirectional encoder representations from transformer architecture (BERT) and embeds industry information during pretraining in two ways: (1) The business communication corpora contain a variety of industry-specific terminology; (2) We employ industry classification (IC) as an additional pretraining objective for text documents originating from companies.

Intended uses & limitations

The model is intended to be fine-tuned on business related NLP tasks, i.e. sequence classification, named entity recognition, sentiment analysis or question answering.

Training data

CompanyWeb: 0.77 billion token, 3.5 GB raw text file
MD&A Disclosures: 1.06 billion token, 5.1 GB raw text file
Semantic Scholar Open Research Corpus: 0.40 billion token, 1.9 GB raw text file

Evaluation results

Classification Tasks

Task	Financial Risk (F1/Acc)	News Headline Topic (F1/Acc)
	85.89/87.02	75.06/67.71

Named Entity Recognition

Task	SEC Filings (F1/Prec/Rec)
	79.82/77.45/83.38

Sentiment Analysis

Task	FiQA (MSE/MAE)	Financial Phrasebank (F1/Acc)	StockTweets (F1/Acc)
	0.0758/0.202	75.06/67.71	69.14/69.54

Question Answering

Task	FinQA (Exe Acc/Prog Acc)
	60.07/57.19

BibTeX entry and citation info

@article{BORCHERT2024,
  title = {Industry-sensitive language modeling for business},
  journal = {European Journal of Operational Research},
  year = {2024},
  issn = {0377-2217},
  doi = {https://doi.org/10.1016/j.ejor.2024.01.023},
  url = {https://www.sciencedirect.com/science/article/pii/S0377221724000444},
  author = {Philipp Borchert and Kristof Coussement and Jochen {De Weerdt} and Arno {De Caigny}},
}

📄 License

This project is licensed under the CC BY 4.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご