Cdlm

🚀 Cross-Document Language Modeling

CDLM is a cross-document language modeling approach, which offers new solutions for language understanding across multiple documents.

🚀 Quick Start

CDLM stands for Cross-Document Language Modeling. It was proposed by Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Peters, Arie Cattan, and Ido Dagan, and presented in EMNLP Findings in 2021. You can access the paper here.

⚠️ Important Note

During the pretraining process, document and sentence separators were used. You may need to add these to your data. The document separators are <doc-s> and </doc-s> (the last two tokens in the vocabulary), and the sentence separators are <s> and </s>.

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModel
# load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained('biu-nlp/cdlm')
model = AutoModel.from_pretrained('biu-nlp/cdlm')

📚 Documentation

The original repository for this project can be found here.

📄 License

This project is licensed under the Apache 2.0 license.

If you find our work useful, please cite the paper as follows:

@article{caciularu2021cross,
  title={Cross-Document Language Modeling},
  author={Caciularu, Avi and Cohan, Arman and Beltagy, Iz and Peters, Matthew E and Cattan, Arie and Dagan, Ido},
  journal={Findings of the Association for Computational Linguistics: EMNLP 2021},
  year={2021}
}