Bigbird Pegasus Large Arxiv

🚀 BigBirdPegasus model (large)

BigBirdPegasus is a sparse - attention based transformer that extends Transformer models to handle much longer sequences, excelling in long - document summarization and other tasks.

🚀 Quick Start

BigBird is a sparse - attention based transformer that extends Transformer - based models like BERT to handle much longer sequences. It comes with a theoretical understanding of what a complete transformer can achieve. BigBird was introduced in this paper and first released in this repository.

Disclaimer: The team releasing BigBird did not write a model card for this model, so this model card has been written by the Hugging Face team.

✨ Features

BigBird relies on block sparse attention instead of normal attention (e.g., BERT's attention). It can handle sequences up to a length of 4096 at a much lower compute cost compared to BERT. It has achieved state - of - the - art (SOTA) results on various tasks involving very long sequences, such as long document summarization and question - answering with long contexts.

💻 Usage Examples

Basic Usage

from transformers import BigBirdPegasusForConditionalGeneration, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/bigbird-pegasus-large-arxiv")

# by default encoder-attention is `block_sparse` with num_random_blocks=3, block_size=64
model = BigBirdPegasusForConditionalGeneration.from_pretrained("google/bigbird-pegasus-large-arxiv")

# decoder attention type can't be changed & will be "original_full"
# you can change `attention_type` (encoder only) to full attention like this:
model = BigBirdPegasusForConditionalGeneration.from_pretrained("google/bigbird-pegasus-large-arxiv", attention_type="original_full")

# you can change `block_size` & `num_random_blocks` like this:
model = BigBirdPegasusForConditionalGeneration.from_pretrained("google/bigbird-pegasus-large-arxiv", block_size=16, num_random_blocks=2)

text = "Replace me by any text you'd like."
inputs = tokenizer(text, return_tensors='pt')
prediction = model.generate(**inputs)
prediction = tokenizer.batch_decode(prediction)

📚 Documentation

This checkpoint is obtained after fine - tuning BigBirdPegasusForConditionalGeneration for summarization on arxiv dataset from scientific_papers.

📄 License

This model is released under the Apache 2.0 license.

📊 Model Metrics

📖 BibTeX entry and citation info

@misc{zaheer2021big,
      title={Big Bird: Transformers for Longer Sequences}, 
      author={Manzil Zaheer and Guru Guruganesh and Avinava Dubey and Joshua Ainslie and Chris Alberti and Santiago Ontanon and Philip Pham and Anirudh Ravula and Qifan Wang and Li Yang and Amr Ahmed},
      year={2021},
      eprint={2007.14062},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}