Led Base 16384 Billsum Summarization

🚀 Longformer Encoder-Decoder (LED) fine-tuned on Billsum

This model is a fine - tuned version of led - base - 16384 on the Billsum dataset. It can effectively handle long - document summarization tasks, leveraging the capabilities of the Longformer architecture.

🚀 Quick Start

Use In Transformers

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Artifact-AI/led_base_16384_billsum_summarization")

model = AutoModelForSeq2SeqLM.from_pretrained("Artifact-AI/led_base_16384_billsum_summarization")

✨ Features

• Fine - tuned on Billsum: The model is specifically fine - tuned on the Billsum dataset, which is suitable for legislative bill summarization.
• Long - document handling: Based on the Longformer architecture, it can process up to 16K tokens, enabling it to handle long legislative bills.

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Artifact-AI/led_base_16384_billsum_summarization")
model = AutoModelForSeq2SeqLM.from_pretrained("Artifact-AI/led_base_16384_billsum_summarization")

input_text = "Your long legislative bill text here"
input_ids = tokenizer(input_text, return_tensors='pt').input_ids
output = model.generate(input_ids)
summary = tokenizer.decode(output[0], skip_special_tokens=True)
print(summary)

📚 Documentation

Model Initialization

As described in Longformer: The Long - Document Transformer by Iz Beltagy, Matthew E. Peters, Arman Cohan, led - base - 16384 was initialized from bart - base since both models share the exact same architecture. To be able to process 16K tokens, bart - base's position embedding matrix was simply copied 16 times.

Results

The model is trained on the BillSum summarization dataset found here.

Test The Model

Please find a notebook to test the model below:

📄 License

No license information provided in the original document.

📚 Citing & Authors

@misc{led_base_16384_billsum_summarization,
    title={

(The citation seems incomplete in the original document. Please provide the full citation if needed.)