Roberta Temporal Predictor

🚀 roberta-temporal-predictor

A RoBERTa-base model fine-tuned on the The New York Times Annotated Corpus to predict the temporal precedence of two events. It serves as the "temporality prediction" component in our ROCK framework for commonsense causality reasoning.

🚀 Quick Start

This RoBERTa-base model is fine-tuned on The New York Times Annotated Corpus to predict the temporal precedence of two events. It is used as the "temporality prediction" component in our ROCK framework for reasoning about commonsense causality. For more details, please refer to our paper.

✨ Features

• Fine-tuned on The New York Times Annotated Corpus for temporal precedence prediction.
• Used as a key component in the ROCK framework for commonsense causality reasoning.

💻 Usage Examples

Basic Usage

You can directly use this model for filling-mask tasks, as shown in the example widget.

# Widget examples
widget:
- text: "The man turned on the faucet <mask> water flows out."
- text: "The woman received her pension <mask> she retired."

Advanced Usage

For better temporal inference, it is recommended to symmetrize the outputs as $$ P(E_1 \prec E_2) = \frac{1}{2} (f(E_1,E_2) + f(E_2,E_1)) $$ where f(E_1,E_2) denotes the predicted probability for E_1 to occur preceding E_2. For simplicity, we implement the following TempPredictor class that incorporate this symmetrization automatically.

from transformers import (RobertaForMaskedLM, RobertaTokenizer)
from src.temp_predictor import TempPredictor

TORCH_DEV = "cuda:0" # change as needed

tp_roberta_ft = src.TempPredictor(
    model=RobertaForMaskedLM.from_pretrained("CogComp/roberta-temporal-predictor"),
    tokenizer=RobertaTokenizer.from_pretrained("CogComp/roberta-temporal-predictor"),
    device=TORCH_DEV
)

E1 = "The man turned on the faucet."
E2 = "Water flows out."
t12 = tp_roberta_ft(E1, E2, top_k=5)
print(f"P('{E1}' before '{E2}'): {t12}")

📄 License

This project is licensed under the MIT license.

📚 Documentation

BibTeX entry and citation info

@misc{zhang2022causal,
      title={Causal Inference Principles for Reasoning about Commonsense Causality}, 
      author={Jiayao Zhang and Hongming Zhang and Dan Roth and Weijie J. Su},
      year={2022},
      eprint={2202.00436},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}