đ Model Card for kobart-base-v2
KoBART-base-v2 is a Korean encoder-decoder language model based on the BART architecture, trained on over 40GB of Korean text.
đ Quick Start
Use the code below to get started with the model.
Click to expand
from transformers import PreTrainedTokenizerFast, BartModel
tokenizer = PreTrainedTokenizerFast.from_pretrained('gogamza/kobart-base-v2')
model = BartModel.from_pretrained('gogamza/kobart-base-v2')
⨠Features
- Feature Extraction: This model can be used for the task of feature extraction.
- Korean Language Support: Specifically designed for the Korean language, trained on a large amount of Korean text.
đĻ Installation
No specific installation steps are provided in the original document.
đģ Usage Examples
Basic Usage
from transformers import PreTrainedTokenizerFast, BartModel
tokenizer = PreTrainedTokenizerFast.from_pretrained('gogamza/kobart-base-v2')
model = BartModel.from_pretrained('gogamza/kobart-base-v2')
đ Documentation
Model Details
BART (Bidirectional and Auto-Regressive Transformers) is trained in the form of an autoencoder by adding noise to parts of the input text and then restoring it to the original text. Korean BART (hereinafter KoBART) is a Korean encoder-decoder language model trained on over 40GB of Korean text using the Text Infilling noise function used in the paper. We are releasing the derived KoBART-base.
- Developed by: More information needed
- Shared by [Optional]: Heewon(Haven) Jeon
- Model type: Feature Extraction
- Language(s) (NLP): Korean
- License: MIT
- Parent Model: BART
- Resources for more information:
Uses
Direct Use
This model can be used for the task of Feature Extraction.
Out-of-Scope Use
The model should not be used to intentionally create hostile or alienating environments for people.
Bias, Risks, and Limitations
Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
Training Details
Training Data
| Data |
# of Sentences |
| Korean Wiki |
5M |
| Other corpus |
0.27B |
In addition to the Korean Wikipedia, various data such as news, books, Modu Corpus v1.0 (conversations, news, ...), and Blue House Petitions were used for model training.
The vocab size is 30,000, and emoticons and emojis commonly used in conversations, such as đ, đ, đ, đ
, đ¤Ŗ, .. , :-), :), -), (-:..., were added to improve the recognition ability of these tokens.
Training Procedure
Tokenizer
It was trained using the Character BPE tokenizer from the tokenizers package.
Speeds, Sizes, Times
| Model |
# of params |
Type |
# of layers |
# of heads |
ffn_dim |
hidden_dims |
KoBART-base |
124M |
Encoder |
6 |
16 |
3072 |
768 |
|
|
Decoder |
6 |
16 |
3072 |
768 |
Evaluation
Testing Data, Factors & Metrics
More information needed for testing data, factors, and metrics.
Results
NSMC
The model authors also note in the GitHub Repo:
Model Examination
More information needed.
Environmental Impact
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
- Hardware Type: More information needed
- Hours used: More information needed
- Cloud Provider: More information needed
- Compute Region: More information needed
- Carbon Emitted: More information needed
Technical Specifications [optional]
More information needed for model architecture, objective, compute infrastructure (hardware and software).
Citation
More information needed for BibTeX citation.
Glossary [optional]
More information needed.
More Information [optional]
More information needed.
Model Card Authors [optional]
Heewon(Haven) Jeon in collaboration with Ezi Ozoani and the Hugging Face team
Model Card Contact
The model authors note in the GitHub Repo:
Please post issues related to KoBART here.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
