Mluke Base

🚀 mLUKE

mLUKE (multilingual LUKE) is a multilingual extension of LUKE. It offers capabilities for named entity recognition, relation classification, and question answering.

This is the mLUKE base model with 12 hidden layers and a hidden size of 768. The total number of parameters in this model is 585M (278M for the word embeddings and encoder, 307M for the entity embeddings). The model was initialized with the weights of XLM - RoBERTa(base) and trained using the December 2020 version of Wikipedia in 24 languages.

Please check the official repository for more details and updates.

✨ Features

• Multilingual support including ar, bn, de, el, en, es, fi, fr, hi, id, it, ja, ko, nl, pl, pt, ru, sv, sw, te, th, tr, vi, zh.
• Capable of named entity recognition, relation classification, and question answering.

📚 Documentation

Note

When you load the model from AutoModel.from_pretrained with the default configuration, you will see the following warning:

Some weights of the model checkpoint at studio-ousia/mluke-base-lite were not used when initializing LukeModel: [
'luke.encoder.layer.0.attention.self.w2e_query.weight', 'luke.encoder.layer.0.attention.self.w2e_query.bias', 
'luke.encoder.layer.0.attention.self.e2w_query.weight', 'luke.encoder.layer.0.attention.self.e2w_query.bias', 
'luke.encoder.layer.0.attention.self.e2e_query.weight', 'luke.encoder.layer.0.attention.self.e2e_query.bias', 
...]

These weights are the weights for entity - aware attention (as described in the LUKE paper). This is expected because use_entity_aware_attention is set to false by default, but the pretrained weights contain the weights for it in case you enable use_entity_aware_attention and have the weights loaded into the model.

Citation

If you find mLUKE useful for your work, please cite the following paper:

@inproceedings{ri-etal-2022-mluke,
    title = "m{LUKE}: {T}he Power of Entity Representations in Multilingual Pretrained Language Models",
    author = "Ri, Ryokan  and
      Yamada, Ikuya  and
      Tsuruoka, Yoshimasa",
    booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    year = "2022",
    url = "https://aclanthology.org/2022.acl-long.505",

📄 License

This project is licensed under the apache - 2.0 license.