Bigbird Pegasus Large Bigpatent

🚀 BigBirdPegasus模型（大模型）

BigBirdPegasus模型是基于稀疏注意力机制的Transformer模型，它将基于Transformer的模型（如BERT）扩展到更长的序列处理中。此外，BigBird还从理论上阐释了稀疏模型所能处理的完整Transformer的能力。

🚀 快速开始

模型描述

BigBird采用块稀疏注意力机制，而非普通的注意力机制（如BERT的注意力机制），与BERT相比，它能够以更低的计算成本处理长度达4096的序列。在涉及超长序列的各种任务中，如长文档摘要生成、长上下文问答等，该模型都达到了当前最优水平。

如何使用

以下是在PyTorch中使用该模型获取给定文本特征的示例代码：

from transformers import BigBirdPegasusForConditionalGeneration, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/bigbird-pegasus-large-bigpatent")

# by default encoder-attention is `block_sparse` with num_random_blocks=3, block_size=64
model = BigBirdPegasusForConditionalGeneration.from_pretrained("google/bigbird-pegasus-large-bigpatent")

# decoder attention type can't be changed & will be "original_full"
# you can change `attention_type` (encoder only) to full attention like this:
model = BigBirdPegasusForConditionalGeneration.from_pretrained("google/bigbird-pegasus-large-bigpatent", attention_type="original_full")

# you can change `block_size` & `num_random_blocks` like this:
model = BigBirdPegasusForConditionalGeneration.from_pretrained("google/bigbird-pegasus-large-bigpatent", block_size=16, num_random_blocks=2)

text = "Replace me by any text you'd like."
inputs = tokenizer(text, return_tensors='pt')
prediction = model.generate(**inputs)
prediction = tokenizer.batch_decode(prediction)

训练过程

此检查点是在 big_patent 数据集上对 BigBirdPegasusForConditionalGeneration 进行摘要生成微调后得到的。

BibTeX引用和引用信息

@misc{zaheer2021big,
      title={Big Bird: Transformers for Longer Sequences}, 
      author={Manzil Zaheer and Guru Guruganesh and Avinava Dubey and Joshua Ainslie and Chris Alberti and Santiago Ontanon and Philip Pham and Anirudh Ravula and Qifan Wang and Li Yang and Amr Ahmed},
      year={2021},
      eprint={2007.14062},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

免责声明

发布BigBird的团队并未为此模型撰写模型卡片，此模型卡片由Hugging Face团队撰写。

信息表格