Bert Political Election2020 Twitter Mlm

🚀 Pre-trained BERT on Twitter US Political Election 2020

This project provides pre-trained weights for stance detection, trained on over 5 million English tweets about the 2020 US Presidential Election.

🚀 Quick Start

This pre-trained language model is initialized with BERT-base and can be fine-tuned for various downstream tasks, such as classification.

✨ Features

• Pre-trained on a large dataset of over 5 million English tweets about the 2020 US Presidential Election.
• Initialized with BERT-base weights and trained with the normal MLM objective.
• Can be fine-tuned for any downstream task.

📦 Installation

To use this model, you need to have the transformers and torch libraries installed. You can install them using the following commands:

pip install transformers torch

💻 Usage Examples

Basic Usage

from transformers import BertTokenizer, BertForMaskedLM, pipeline
import torch

# Choose GPU if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Select mode path here
pretrained_LM_path = "kornosk/bert-political-election2020-twitter-mlm"

# Load model
tokenizer = BertTokenizer.from_pretrained(pretrained_LM_path)
model = BertForMaskedLM.from_pretrained(pretrained_LM_path)

# Fill mask
example = "Trump is the [MASK] of USA"
fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)
# Use following line instead of the above one does not work.
# Huggingface have been updated, newer version accepts a string of model name instead.
fill_mask = pipeline('fill-mask', model=pretrained_LM_path, tokenizer=tokenizer)

outputs = fill_mask(example)
print(outputs)

# See embeddings
inputs = tokenizer(example, return_tensors="pt")
outputs = model(**inputs)
print(outputs)

# OR you can use this model to train on your downstream task!
# Please consider citing our paper if you feel this is useful :)

📚 Documentation

For more detailed information, please refer to the official repository.

📄 License

This project is licensed under the GPL-3.0 license.

📚 Reference

• Knowledge Enhance Masked Language Model for Stance Detection, NAACL 2021.

📚 Citation

@inproceedings{kawintiranon2021knowledge,
    title={Knowledge Enhanced Masked Language Model for Stance Detection},
    author={Kawintiranon, Kornraphop and Singh, Lisa},
    booktitle={Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
    year={2021},
    publisher={Association for Computational Linguistics},
    url={https://www.aclweb.org/anthology/2021.naacl-main.376}
}