Prophetnet Large Uncased

🚀 ProphetNet - Large Uncased

Pretrained weights for ProphetNet, a novel pre - trained language model for sequence - to - sequence learning.

🚀 Quick Start

ProphetNet is a new pre - trained language model designed for sequence - to - sequence learning. It comes with a unique self - supervised objective known as future n - gram prediction. This model can predict multiple future tokens using a n - stream decoder. The original implementation of ProphetNet is in the Fairseq version, available at the github repo.

💻 Usage Examples

Basic Usage

# This pre-trained model can be fine-tuned on *sequence-to-sequence* tasks.
# The model could *e.g.* be trained on headline generation as follows:
from transformers import ProphetNetForConditionalGeneration, ProphetNetTokenizer

model = ProphetNetForConditionalGeneration.from_pretrained("microsoft/prophetnet-large-uncased")
tokenizer = ProphetNetTokenizer.from_pretrained("microsoft/prophetnet-large-uncased")

input_str = "the us state department said wednesday it had received no formal word from bolivia that it was expelling the us ambassador there but said the charges made against him are `` baseless ."
target_str = "us rejects charges against its ambassador in bolivia"

input_ids = tokenizer(input_str, return_tensors="pt").input_ids
labels = tokenizer(target_str, return_tensors="pt").input_ids

loss = model(input_ids, labels=labels).loss

📄 License

Citation

@article{yan2020prophetnet,
  title={Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training},
  author={Yan, Yu and Qi, Weizhen and Gong, Yeyun and Liu, Dayiheng and Duan, Nan and Chen, Jiusheng and Zhang, Ruofei and Zhou, Ming},
  journal={arXiv preprint arXiv:2001.04063},
  year={2020}
}