Long T5 Tglobal Base 16384 Booksum V11 Big Patent V2

🚀 BigBird - Transformer Model for Long Sequences

BigBird is a transformer-based model that addresses the computational limitations of traditional transformers when dealing with long sequences. It offers a more efficient way to handle tasks such as long document summarization and question-answering.

🚀 Quick Start

BigBird is designed to handle long sequences more efficiently compared to traditional transformer-based models. It uses block sparse attention instead of normal attention, which allows it to process sequences up to a length of 4096 at a much lower computational cost.

✨ Features

• Low Computational Cost: BigBird can handle long sequences (up to 4096) with significantly lower computational cost compared to models like BERT.
• SOTA Performance: It has achieved state-of-the-art results on various tasks involving long sequences, such as long document summarization and question-answering with long contexts.
• Efficient Attention Mechanism: The block sparse attention mechanism approximates the full attention matrix, making it more efficient for long sequences.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

# let's consider following sentence as an example
example = ['BigBird', 'is', 'now', 'available', 'in', 'HuggingFace', 'for', 'extractive', 'question', 'answering']

# further let's assume, we're trying to understand the representation of 'available' i.e.
query_token = 'available'
# We will initialize an empty `set` and fill up the tokens of our interest as we proceed in this section.
key_tokens = []

Advanced Usage

# There is no specific advanced usage example provided in the original document.

📚 Documentation

Transformer-based models have shown great potential in many NLP tasks. However, their O(n^2) time and memory complexity (where n is the sequence length) makes them computationally expensive for long sequences (n > 512). BigBird addresses this issue by using block sparse attention.

Some of the key questions when working with standard BERT-like attention include:

• Do all tokens really have to attend to all other tokens?
• Why not compute attention only over important tokens?
• How to decide what tokens are important?
• How to attend to just a few tokens in a very efficient way?

In the context of BigBird, we can consider a practical example of how attention works. For instance, in the sentence "BigBird is now available in HuggingFace for extractive question answering", in BERT-like attention, every word would attend to all other tokens. But with BigBird's approach, we can be more selective about which tokens a queried token should attend to.

🔧 Technical Details

BigBird relies on block sparse attention instead of normal attention (i.e., BERT's attention). This allows it to approximate the full attention matrix and handle sequences up to a length of 4096 at a much lower computational cost compared to BERT.

The attention mechanism in BigBird is designed to address the limitations of traditional transformers when dealing with long sequences. By using block sparse attention, it can focus on important tokens and reduce the computational burden.

📄 License

• Apache-2.0
• BSD-3-Clause