Longformer Base 4096 Spanish

🚀 longformer-base-4096-spanish

Longformer is a Transformer model designed for processing long documents, offering extended sequence length support and a unique attention mechanism.

🚀 Quick Start

Longformer, as introduced in this paper, is a Transformer model tailored for long documents. The longformer-base-4096 is a BERT - like model. It initiates from the RoBERTa checkpoint (specifically, BERTIN in this instance) and undergoes pre - training for Masked Language Modeling (MLM) on long documents sourced from BETO's all_wikis. Notably, it can handle sequences up to a length of 4,096!

Longformer employs a combination of sliding window (local) attention and global attention. The global attention can be configured by the user according to the specific task, enabling the model to learn task - specific representations.

This model is developed based on the research conducted by Iz Beltagy, Matthew E. Peters, and Arman Cohan.

📄 License

This project is licensed under the MIT license.

📚 Documentation

Tags

• Long documents
• longformer
• bertin
• spanish

Datasets

• spanish_large_corpus

Widget

• "Manuel Romero ha creado con el equipo de BERTIN un modelo que procesa documentos largos."

Language

• Spanish

📚 Citation

If you wish to cite this model, you can use the following BibTeX entry:

@misc{mromero2022longformer-base-4096-spanish,
  title={Spanish LongFormer by Manuel Romero},
  author={Romero, Manuel},
  publisher={Hugging Face},
  journal={Hugging Face Hub},
  howpublished={\url{https://huggingface.co/mrm8488/longformer-base-4096-spanish}},
  year={2022}
}