đ EconoBert
This model, EconoBert, is a fine - tuned version of the pre - trained bert - base - uncased model. It is specifically fine - tuned on an economics - related dataset, aiming to provide better performance for NLP tasks in the fields of economics, politics, and finance.
đ Quick Start
This model can be used as a backbone for NLP tasks in the domains of economics, politics, and finance. You can load it using relevant NLP libraries and start fine - tuning or making predictions.
⨠Features
- Fine - tuned on Economics Data: It is fine - tuned on a dataset specific to the economics domain, making it more suitable for related NLP tasks.
- Good Performance: Achieves an accuracy of 73% for the MLM task and 95% for the NSP task on the test set.
đĻ Installation
No specific installation steps are provided in the original document.
đģ Usage Examples
No code examples are provided in the original document.
đ Documentation
Model description
The model is a simple fine - tuning of a base BERT model on a dataset specific to the domain of economics. It follows the same architecture as the base model, and no resize_token_embeddings operation was required.
Intended uses & limitations
This model should be used as a backbone for NLP tasks applied to the domains of economics, politics, and finance.
Training and evaluation data
The dataset used for fine - tuning is samchain/BIS_Speeches_97_23. It consists of 773k pairs of sentences, with half being negative pairs (sequence A and B are not related) and the other half being positive pairs (sequence B follows sequence A). The test set is made up of 136k pairs.
Training procedure
The model was fine - tuned for 2 epochs, with a batch size of 64 and a sequence length of 128. The Adam optimizer was used with a learning rate of 1e - 5.
Training hyperparameters
| Property |
Details |
| Optimizer |
{'name': 'Adam', 'weight_decay': None, 'clipnorm': None, 'global_clipnorm': None, 'clipvalue': None, 'use_ema': False, 'ema_momentum': 0.99, 'ema_overwrite_frequency': None, 'jit_compile': True, 'is_legacy_optimizer': False, 'learning_rate': 1e - 05, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e - 07, 'amsgrad': False} |
| Training Precision |
float32 |
Training results
The training loss is 1.6046 on the training set and 1.47 on the test set.
Framework versions
| Property |
Details |
| Transformers |
4.31.0 |
| TensorFlow |
2.12.0 |
| Datasets |
2.13.1 |
| Tokenizers |
0.13.3 |
đ§ Technical Details
The model is a fine - tuned version of bert - base - uncased. It uses the same architecture as the base model and is trained on a specific economics - related dataset. The training process involves fine - tuning for 2 epochs with specific hyperparameters such as batch size, sequence length, and optimizer settings.
đ License
This model is released under the Apache 2.0 license.
Citing & Authors
Samuel Chaineau
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
