Luke-large Open-Source Pretrained Model - Providing Deep Contextual Representations for Words and Entities

Luke Large

Developed by studio-ousia

LUKE is a Transformer-based pre-trained model specifically designed for words and entities, providing deep contextual representations through entity-aware self-attention mechanisms.

Large Language Model

Transformers

EnglishOpen Source License:Apache-2.0 #Entity-aware Attention #Multi-task NLP #Knowledge-enhanced Representation

Downloads 1,040

Release Time : 3/2/2022

Model Overview

LUKE is an innovative pre-trained contextual representation method that treats words and entities in text as independent tokens and outputs their context-dependent representations. The model employs an entity-aware self-attention mechanism, extending the traditional Transformer's self-attention by considering token types (word or entity) when computing attention scores.

Model Features

Entity-aware Self-attention Mechanism

Extends the traditional Transformer's self-attention mechanism by considering token types (word or entity) when computing attention scores.

Joint Representation of Words and Entities

Treats words and entities in text as independent tokens and outputs their context-dependent representations.

Outstanding Multi-task Performance

Achieves state-of-the-art results on five mainstream natural language processing benchmarks.

Model Capabilities

Named Entity Recognition

Entity Typing

Relation Classification

Extractive Question Answering

Cloze-style Question Answering

Use Cases

Information Extraction

Named Entity Recognition

Identify and classify named entities (e.g., person names, locations, organizations) from text

Achieves 94.3 F1 score on the CoNLL-2003 dataset

Relation Classification

Identify relationship types between entities

Achieves 72.7 F1 score on the TACRED dataset

Question Answering

Extractive Question Answering

Extract answers from given text to answer natural language questions

Achieves 90.2 EM/95.4 F1 on the SQuAD v1.1 dataset

Cloze-style Question Answering

Fill in blanks in sentences by understanding context

Achieves 90.6 EM/91.2 F1 on the ReCoRD dataset

🚀 LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

LUKE (Language Understanding with Knowledge-based Embeddings) is a novel pre-trained contextualized representation for words and entities, built upon the Transformer architecture. It treats words and entities in a given text as independent tokens and outputs their contextualized representations. LUKE employs an entity-aware self-attention mechanism, which extends the self-attention mechanism of the Transformer, and takes into account the token types (words or entities) when calculating attention scores.

LUKE has achieved state-of-the-art results on five popular NLP benchmarks, including SQuAD v1.1 (extractive question answering), CoNLL-2003 (named entity recognition), ReCoRD (cloze-style question answering), TACRED (relation classification), and Open Entity (entity typing).

For more details and updates, please visit the official repository.

This is the LUKE large model with 24 hidden layers and a hidden size of 1024. The total number of parameters in this model is 483M. It was trained using the December 2018 version of Wikipedia.

✨ Features

Treats words and entities as independent tokens.
Adopts an entity-aware self-attention mechanism.
Achieves state-of-the-art results on multiple NLP benchmarks.

📚 Documentation

Experimental results

The experimental results are presented in the following table:

Task	Dataset	Metric	LUKE-large	luke-base	Previous SOTA
Extractive Question Answering	SQuAD v1.1	EM/F1	90.2/95.4	86.1/92.3	89.9/95.1 (Yang et al., 2019)
Named Entity Recognition	CoNLL-2003	F1	94.3	93.3	93.5 (Baevski et al., 2019)
Cloze-style Question Answering	ReCoRD	EM/F1	90.6/91.2	-	83.1/83.7 (Li et al., 2019)
Relation Classification	TACRED	F1	72.7	-	72.0 (Wang et al. , 2020)
Fine-grained Entity Typing	Open Entity	F1	78.2	-	77.6 (Wang et al. , 2020)

Citation

If you find LUKE useful for your work, please cite the following paper:

@inproceedings{yamada2020luke,
  title={LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention},
  author={Ikuya Yamada and Akari Asai and Hiroyuki Shindo and Hideaki Takeda and Yuji Matsumoto},
  booktitle={EMNLP},
  year={2020}
}

📄 License

This project is licensed under the Apache-2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご