Bigbird Pegasus Large Bigpatent

🚀 BigBirdPegasus模型（大模型）

BigBirdPegasus模型是基於稀疏注意力機制的Transformer模型，它將基於Transformer的模型（如BERT）擴展到更長的序列處理中。此外，BigBird還從理論上闡釋了稀疏模型所能處理的完整Transformer的能力。

🚀 快速開始

模型描述

BigBird採用塊稀疏注意力機制，而非普通的注意力機制（如BERT的注意力機制），與BERT相比，它能夠以更低的計算成本處理長度達4096的序列。在涉及超長序列的各種任務中，如長文檔摘要生成、長上下文問答等，該模型都達到了當前最優水平。

如何使用

以下是在PyTorch中使用該模型獲取給定文本特徵的示例代碼：

from transformers import BigBirdPegasusForConditionalGeneration, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/bigbird-pegasus-large-bigpatent")

# by default encoder-attention is `block_sparse` with num_random_blocks=3, block_size=64
model = BigBirdPegasusForConditionalGeneration.from_pretrained("google/bigbird-pegasus-large-bigpatent")

# decoder attention type can't be changed & will be "original_full"
# you can change `attention_type` (encoder only) to full attention like this:
model = BigBirdPegasusForConditionalGeneration.from_pretrained("google/bigbird-pegasus-large-bigpatent", attention_type="original_full")

# you can change `block_size` & `num_random_blocks` like this:
model = BigBirdPegasusForConditionalGeneration.from_pretrained("google/bigbird-pegasus-large-bigpatent", block_size=16, num_random_blocks=2)

text = "Replace me by any text you'd like."
inputs = tokenizer(text, return_tensors='pt')
prediction = model.generate(**inputs)
prediction = tokenizer.batch_decode(prediction)

訓練過程

此檢查點是在 big_patent 數據集上對 BigBirdPegasusForConditionalGeneration 進行摘要生成微調後得到的。

BibTeX引用和引用信息

@misc{zaheer2021big,
      title={Big Bird: Transformers for Longer Sequences}, 
      author={Manzil Zaheer and Guru Guruganesh and Avinava Dubey and Joshua Ainslie and Chris Alberti and Santiago Ontanon and Philip Pham and Anirudh Ravula and Qifan Wang and Li Yang and Amr Ahmed},
      year={2021},
      eprint={2007.14062},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

免責聲明

發佈BigBird的團隊並未為此模型撰寫模型卡片，此模型卡片由Hugging Face團隊撰寫。

信息表格