Mluke Large

🚀 mLUKE

mLUKE (multilingual LUKE) is a multilingual extension of LUKE, which can be applied in various NLP tasks such as named entity recognition, relation classification, and question answering.

🚀 Quick Start

Please check the official repository for more details and updates.

✨ Features

• Multilingual Support: Supports multiple languages including Arabic, Bengali, German, Greek, English, Spanish, Finnish, French, Hindi, Indonesian, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Russian, Swedish, Swahili, Telugu, Thai, Turkish, Vietnamese, and Chinese.
• Large Model: This is the mLUKE large model with 24 hidden layers and a 768 hidden size. The total number of parameters in this model is 868M (561M for the word embeddings and encoder, 307M for the entity embeddings).
• Pretrained with XLM - RoBERTa: The model was initialized with the weights of XLM - RoBERTa(large) and trained using the December 2020 version of Wikipedia in 24 languages.

📚 Documentation

Note

When you load the model from AutoModel.from_pretrained with the default configuration, you will see the following warning:

Some weights of the model checkpoint at studio-ousia/mluke-base-lite were not used when initializing LukeModel: [
'luke.encoder.layer.0.attention.self.w2e_query.weight', 'luke.encoder.layer.0.attention.self.w2e_query.bias', 
'luke.encoder.layer.0.attention.self.e2w_query.weight', 'luke.encoder.layer.0.attention.self.e2w_query.bias', 
'luke.encoder.layer.0.attention.self.e2e_query.weight', 'luke.encoder.layer.0.attention.self.e2e_query.bias', 
...]

These weights are the weights for entity - aware attention (as described in the LUKE paper). This is expected because use_entity_aware_attention is set to false by default, but the pretrained weights contain the weights for it in case you enable use_entity_aware_attention and have the weights loaded into the model.

Citation

If you find mLUKE useful for your work, please cite the following paper:

@inproceedings{ri-etal-2022-mluke,
    title = "m{LUKE}: {T}he Power of Entity Representations in Multilingual Pretrained Language Models",
    author = "Ri, Ryokan  and
      Yamada, Ikuya  and
      Tsuruoka, Yoshimasa",
    booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    year = "2022",
    url = "https://aclanthology.org/2022.acl-long.505",

📄 License

This project is licensed under the apache - 2.0 license.