Luke-base Open-source Pre-trained Model - Providing Deep Contextual Representations for Words and Entities

Luke Base

Developed by studio-ousia

LUKE is a Transformer-based pre-trained model specifically designed for words and entities, providing deep contextual representations through entity-aware self-attention mechanisms.

Large Language Model

Transformers

EnglishOpen Source License:Apache-2.0 #Entity-aware Attention #Multi-task NLP #Knowledge-enhanced Representations

Downloads 2,358

Release Time : 3/2/2022

Model Overview

LUKE treats words and entities in text as independent tokens and outputs their context-dependent representations. The model employs an entity-aware self-attention mechanism that extends the Transformer's self-attention mechanism by considering token types (word or entity) when computing attention scores.

Model Features

Entity-aware Self-attention Mechanism

Extends the Transformer's self-attention mechanism by considering token types (word or entity) when computing attention scores.

Deep Contextual Representations

Provides context-dependent representations for words and entities, suitable for various NLP tasks.

Multi-task Support

Performs excellently in multiple tasks including named entity recognition, entity typing, relation classification, and question answering.

Model Capabilities

Named Entity Recognition

Entity Typing

Relation Classification

Question Answering

Use Cases

Natural Language Processing

Named Entity Recognition

Identify named entities in text (e.g., person names, locations, organizations).

Achieved F1 score of 94.3 on CoNLL-2003 dataset, surpassing previous best result of 93.5.

Relation Classification

Identify relationships between entities.

Achieved F1 score of 72.7 on TACRED dataset, surpassing previous best result of 72.0.

Question Answering

Answer text-based questions.

Achieved EM/F1 scores of 90.2/95.4 on SQuAD v1.1 dataset, surpassing previous best results of 89.9/95.1.

🚀 LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

LUKE (Language Understanding with Knowledge-based Embeddings) is a novel pre-trained contextualized representation for words and entities, built upon the transformer architecture. It treats words and entities in a given text as independent tokens and outputs their contextualized representations. LUKE employs an entity-aware self-attention mechanism, which extends the self-attention mechanism of the transformer and takes into account the token types (words or entities) when calculating attention scores.

LUKE achieves state-of-the-art performance on five prominent NLP benchmarks, including SQuAD v1.1 (extractive question answering), CoNLL-2003 (named entity recognition), ReCoRD (cloze-style question answering), TACRED (relation classification), and Open Entity (entity typing).

For more details and updates, please visit the official repository.

This is the LUKE base model with 12 hidden layers and a hidden size of 768. The model has a total of 253M parameters and is trained using the December 2018 version of Wikipedia.

✨ Features

Treats words and entities as independent tokens and outputs their contextualized representations.
Employs an entity-aware self-attention mechanism to consider token types when computing attention scores.
Achieves state-of-the-art results on multiple NLP benchmarks.

📚 Documentation

Experimental results

The experimental results are presented in the following table:

Task	Dataset	Metric	LUKE-large	luke-base	Previous SOTA
Extractive Question Answering	SQuAD v1.1	EM/F1	90.2/95.4	86.1/92.3	89.9/95.1 (Yang et al., 2019)
Named Entity Recognition	CoNLL-2003	F1	94.3	93.3	93.5 (Baevski et al., 2019)
Cloze-style Question Answering	ReCoRD	EM/F1	90.6/91.2	-	83.1/83.7 (Li et al., 2019)
Relation Classification	TACRED	F1	72.7	-	72.0 (Wang et al. , 2020)
Fine-grained Entity Typing	Open Entity	F1	78.2	-	77.6 (Wang et al. , 2020)

Citation

If you find LUKE useful for your work, please cite the following paper:

@inproceedings{yamada2020luke,
  title={LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention},
  author={Ikuya Yamada and Akari Asai and Hiroyuki Shindo and Hideaki Takeda and Yuji Matsumoto},
  booktitle={EMNLP},
  year={2020}
}

📄 License

This project is licensed under the Apache-2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご