Long-t5-tglobal-xl open-source text processing model - Free deployment supports long-sequence text input

Long T5 Tglobal Xl

Developed by google

LongT5 is a Transformer-based text-to-text model specifically designed to handle long sequence inputs, supporting up to 16,384 tokens.

Large Language Model

Transformers

EnglishOpen Source License:Apache-2.0 #Long sequence processing #Transient global attention #Text generation

Downloads 336

Release Time : 6/14/2022

Model Overview

LongT5 is an extended version of the T5 model, adopting an encoder-decoder architecture. It efficiently processes long sequence texts through the transient global attention mechanism and is particularly suitable for tasks that require handling long inputs, such as summary generation and question answering.

Model Features

Transient global attention

Efficiently processes long sequences through a sparse attention pattern, supporting up to 16,384 tokens of input.

Text-to-text framework

Adopts a unified text-to-text conversion framework, supporting multiple generation tasks.

Pegasus-based pre-training

Uses a denoising generative pre-training method similar to Pegasus.

Model Capabilities

Long text summary generation

Long text question answering

Text denoising generation

Use Cases

Text summarization

Long document summary

Generate a concise summary from long documents (such as research reports or news articles).

Question answering system

Long context question answering

Answer complex questions based on the content of long documents.

🚀 LongT5 (transient-global attention, XL-sized model)

The LongT5 model is pre-trained on the English language, offering efficient text-to-text transformation for long sequences.

🚀 Quick Start

The LongT5 model is pre-trained on the English language. It was introduced in the paper LongT5: Efficient Text-To-Text Transformer for Long Sequences by Guo et al. and first released in the LongT5 repository. All the model architecture and configuration can be found in Flaxformer repository which uses another Google research project repository T5x.

Disclaimer: The team releasing LongT5 did not write a model card for this model so this model card has been written by the Hugging Face team.

✨ Features

The LongT5 model is an encoder - decoder transformer pre - trained in a text - to - text denoising generative setting (Pegasus - like generation pre - training).
It is an extension of T5 model, enabling the use of one of two different efficient attention mechanisms: (1) Local attention, or (2) Transient - Global attention. The use of attention sparsity patterns allows the model to efficiently handle input sequences.
It is particularly effective when fine - tuned for text generation (summarization, question answering) which requires handling long input sequences (up to 16,384 tokens).

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, LongT5Model

tokenizer = AutoTokenizer.from_pretrained("google/long-t5-tglobal-xl")
model = LongT5Model.from_pretrained("google/long-t5-tglobal-xl")

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state

📚 Documentation

Intended uses & limitations

The model is mostly meant to be fine - tuned on a supervised dataset. See the model hub to look for fine - tuned versions on a task that interests you.

BibTeX entry and citation info

@article{guo2021longt5,
  title={LongT5: Efficient Text-To-Text Transformer for Long Sequences},
  author={Guo, Mandy and Ainslie, Joshua and Uthus, David and Ontanon, Santiago and Ni, Jianmo and Sung, Yun-Hsuan and Yang, Yinfei},
  journal={arXiv preprint arXiv:2112.07916},
  year={2021}
}

📄 License

This model is licensed under the Apache-2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご