Vlt5 Base Keywords

🚀 Keyword Extraction from Short Texts with T5

Our vlT5 model is a keyword generation model based on an encoder - decoder architecture using Transformer blocks. It can generate precise keyphrases for scientific articles based on their abstracts and titles.

🚀 Quick Start

The vlT5 model offers a powerful solution for keyword extraction from short texts. It leverages an encoder - decoder architecture with Transformer blocks, trained on a scientific articles corpus to predict keyphrases.

✨ Features

• Transferability: The vlT5 model works well across all domains and text types.
• Dual - mode operation: It can work both extractively and abstractively.
• Multilingual support: While trained on Polish and English, it performs relatively well with other languages too.

📦 Installation

No specific installation steps are provided in the original README.

💻 Usage Examples

Basic Usage

from transformers import T5Tokenizer, T5ForConditionalGeneration

model = T5ForConditionalGeneration.from_pretrained("Voicelab/vlt5-base-keywords")
tokenizer = T5Tokenizer.from_pretrained("Voicelab/vlt5-base-keywords")

task_prefix = "Keywords: "
inputs = [
    "Christina Katrakis, who spoke to the BBC from Vorokhta in western Ukraine, relays the account of one family, who say Russian soldiers shot at their vehicles while they were leaving their village near Chernobyl in northern Ukraine. She says the cars had white flags and signs saying they were carrying children.",
    "Decays the learning rate of each parameter group by gamma every step_size epochs. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr.",
    "Hello, I'd like to order a pizza with salami topping.",
]

for sample in inputs:
    input_sequences = [task_prefix + sample]
    input_ids = tokenizer(
        input_sequences, return_tensors="pt", truncation=True
    ).input_ids
    output = model.generate(input_ids, no_repeat_ngram_size=3, num_beams=4)
    predicted = tokenizer.decode(output[0], skip_special_tokens=True)
    print(sample, "\n --->", predicted)

📚 Documentation

vlT5

The biggest advantage of the vlT5 model is its transferability, performing well on all domains and text types. However, the text length and the number of keywords are similar to the training data. For longer texts, they need to be split into smaller chunks before being fed into the model.

Overview

Corpus

The model was trained on a POSMAC corpus. The Polish Open Science Metadata Corpus (POSMAC) is a collection of 216,214 abstracts of scientific publications compiled in the CURLICAT project.

Tokenizer

As in the original plT5 implementation, the training dataset was tokenized into subwords using a sentencepiece unigram model with a vocabulary size of 50k tokens.

Inference

Our results showed that the best generation results were achieved with no_repeat_ngram_size = 3, num_beams = 4

Results

📄 License

CC BY 4.0

🔧 Technical Details

The model uses an encoder - decoder architecture with Transformer blocks. The training dataset was tokenized using a sentencepiece unigram model with a vocabulary size of 50k tokens.

📖 Citation

If you use this model, please cite the following paper: [Pęzik, P., Mikołajczyk, A., Wawrzyński, A., Żarnecki, F., Nitoń, B., Ogrodniczuk, M. (2023). Transferable Keyword Extraction and Generation with Text - to - Text Language Models. In: Mikyška, J., de Mulatier, C., Paszynski, M., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M. (eds) Computational Science – ICCS 2023. ICCS 2023. Lecture Notes in Computer Science, vol 14074. Springer, Cham. https://doi.org/10.1007/978 - 3 - 031 - 36021 - 3_42](https://link.springer.com/chapter/10.1007/978 - 3 - 031 - 36021 - 3_42)

OR

Piotr Pęzik, Agnieszka Mikołajczyk - Bareła, Adam Wawrzyński, Bartłomiej Nitoń, Maciej Ogrodniczuk, Keyword Extraction from Short Texts with a Text - To - Text Transfer Transformer, ACIIDS 2022

👥 Authors

The model was trained by the NLP Research Team at Voicelab.ai. You can contact us here.