đ ccdv/lsg-bart-base-4096-multinews
This model is a fine - tuned version for text summarization, achieving good results on the multi_news dataset.
đ Quick Start
Note:
- Transformers >= 4.36.1
- This model relies on a custom modeling file, you need to add
trust_remote_code=True
- See #13467
LSG ArXiv paper.
Github/conversion script is available at this link.
đģ Usage Examples
Basic Usage
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, pipeline
tokenizer = AutoTokenizer.from_pretrained("ccdv/lsg-bart-base-4096-multinews", trust_remote_code=True)
model = AutoModelForSeq2SeqLM.from_pretrained("ccdv/lsg-bart-base-4096-multinews", trust_remote_code=True)
text = "Replace by what you want."
pipe = pipeline("text2text-generation", model=model, tokenizer=tokenizer, device=0)
generated_text = pipe(
text,
truncation=True,
max_length=64,
no_repeat_ngram_size=7,
num_beams=2,
early_stopping=True
)
⨠Features
This model is a fine - tuned version of ccdv/lsg-bart-base-4096 on the multi_news default dataset. It achieves the following results on the test set:
Test Results with Larger Block Size
Length |
Sparse Type |
Block Size |
Sparsity |
Connexions |
R1 |
R2 |
RL |
RLsum |
4096 |
Local |
256 |
0 |
768 |
47.10 |
18.94 |
25.22 |
43.13 |
4096 |
Local |
128 |
0 |
384 |
46.73 |
18.79 |
25.13 |
42.76 |
4096 |
Pooling |
128 |
4 |
644 |
46.83 |
18.87 |
25.23 |
42.86 |
4096 |
Stride |
128 |
4 |
644 |
46.83 |
18.68 |
24.98 |
42.88 |
4096 |
Block Stride |
128 |
4 |
644 |
46.83 |
18.72 |
25.06 |
42.88 |
4096 |
Norm |
128 |
4 |
644 |
46.74 |
18.60 |
24.93 |
42.79 |
4096 |
LSH |
128 |
4 |
644 |
46.74 |
18.82 |
25.19 |
42.77 |
Test Results with Smaller Block Size (Lower Resources)
Length |
Sparse Type |
Block Size |
Sparsity |
Connexions |
R1 |
R2 |
RL |
RLsum |
4096 |
Local |
64 |
0 |
192 |
45.61 |
17.91 |
24.54 |
41.65 |
4096 |
Local |
32 |
0 |
96 |
43.50 |
16.36 |
23.45 |
39.61 |
4096 |
Pooling |
32 |
4 |
160 |
44.77 |
17.31 |
24.16 |
40.86 |
4096 |
Stride |
32 |
4 |
160 |
45.29 |
17.81 |
24.45 |
41.40 |
4096 |
Block Stride |
32 |
4 |
160 |
45.39 |
17.86 |
24.51 |
41.43 |
4096 |
Norm |
32 |
4 |
160 |
44.65 |
17.25 |
24.09 |
40.76 |
4096 |
LSH |
32 |
4 |
160 |
44.44 |
17.20 |
24.00 |
40.57 |
đ Documentation
Model description
The model relies on Local - Sparse - Global attention to handle long sequences:

The model has about ~145 millions parameters (6 encoder layers - 6 decoder layers).
The model is warm started from BART - base, converted to handle long sequences (encoder only) and fine tuned.
Training and Evaluation
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 8e - 05
- train_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon = 1e - 08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 12.0
Generate hyperparameters
The following hyperparameters were used during generation:
- dataset_name: multi_news
- dataset_config_name: default
- eval_batch_size: 8
- eval_samples: 5622
- early_stopping: True
- ignore_pad_token_for_loss: True
- length_penalty: 2.0
- max_length: 320
- min_length: 32
- num_beams: 5
- no_repeat_ngram_size: None
- seed: 123
Framework versions
- Transformers 4.18.0
- Pytorch 1.10.1+cu102
- Datasets 2.1.0
- Tokenizers 0.11.6
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
đ License
No license information provided in the original document.