RoBERTa-large ConceptNet Mask Prompt E-NCE Open-source Model - Empowering Lexical Relationship Understanding and Analogical Reasoning Tasks

Roberta Large Conceptnet Mask Prompt E Nce

Developed by research-backup

A relation embedding model fine-tuned on RoBERTa-large, specializing in lexical relation understanding and analogical reasoning tasks

Knowledge Graph

Transformers

#Lexical Relation Embedding #High-Precision Relation Classification #Analogical Reasoning

Downloads 17

Release Time : 8/7/2022

Model Overview

This model is fine-tuned on high-confidence ConceptNet data based on RoBERTa-large, specifically designed for understanding lexical relations and performing analogical reasoning. Supports various relation classification and mapping tasks.

Model Features

High-Precision Relation Classification

Achieves F1 scores above 0.9 on multiple relation classification datasets

Multi-Task Support

Simultaneously supports various tasks including relation classification, analogical reasoning, and relation mapping

Prompt-Based Fine-Tuning

Trained using special mask prompt templates to enhance relation understanding capabilities

Model Capabilities

Lexical Relation Classification

Analogical Reasoning

Relation Embedding Generation

Relation Mapping

Use Cases

Natural Language Processing

Lexical Relation Classification

Identify semantic relations between words (e.g., synonymy, antonymy, hypernymy)

Achieves 93% F1 score on the BLESS dataset

Analogical Reasoning

Solve analogy problems like 'Tokyo is to Japan as Paris is to?'

Achieves 89.8% accuracy on the Google Analogy dataset

Knowledge Graph

Relation Mapping

Align and map relations across different knowledge bases

Achieves 93.25% accuracy in relation mapping

🚀 relbert/roberta-large-conceptnet-mask-prompt-e-nce

RelBERT fine-tuned from roberta-large to enhance relation understanding in natural language processing.

🚀 Quick Start

RelBERT is fine-tuned from roberta-large on relbert/conceptnet_high_confidence. The fine-tuning process is carried out via the RelBERT library. For more details, please refer to the repository.

This model achieves the following results on relation understanding tasks:

Analogy Question:
- Dataset, Full Result
- Accuracy on SAT (full): 0.5561497326203209
- Accuracy on SAT: 0.5578635014836796
- Accuracy on BATS: 0.7593107281823235
- Accuracy on U2: 0.5657894736842105
- Accuracy on U4: 0.5902777777777778
- Accuracy on Google: 0.898
Lexical Relation Classification:
- Dataset, Full Result
- Micro F1 score on BLESS: 0.9303902365526593
- Micro F1 score on CogALexV: 0.8781690140845071
- Micro F1 score on EVALution: 0.6939328277356447
- Micro F1 score on K&H+N: 0.9631355637476525
- Micro F1 score on ROOT09: 0.9088060169225948
Relation Mapping:
- Dataset, Full Result
- Accuracy on Relation Mapping: 0.9325396825396826

📦 Installation

This model can be used through the relbert library. Install the library via pip:

pip install relbert

💻 Usage Examples

Basic Usage

from relbert import RelBERT
model = RelBERT("relbert/roberta-large-conceptnet-mask-prompt-e-nce")
vector = model.get_embedding(['Tokyo', 'Japan'])  # shape of (1024, )

📚 Documentation

Training hyperparameters

The following hyperparameters were used during training:

Property	Details
Model	roberta-large
Max Length	64
Mode	mask
Data	relbert/conceptnet_high_confidence
Template Mode	manual
Template	I wasn’t aware of this relationship, but I just read in the encyclopedia that is ’s
Loss Function	nce_logout
Temperature NCE Constant	0.05
Temperature NCE Rank	{'min': 0.01, 'max': 0.05, 'type': 'linear'}
Epoch	146
Batch	128
Learning Rate	5e-06
Learning Rate Decay	False
Learning Rate Warmup	1
Weight Decay	0
Random Seed	0
Exclude Relation	None
Number of Samples	640
Gradient Accumulation	8

The full configuration can be found at fine-tuning parameter file.

📄 License

Reference

If you use any resource from RelBERT, please consider citing our paper.

@inproceedings{ushio-etal-2021-distilling-relation-embeddings,
    title = "{D}istilling {R}elation {E}mbeddings from {P}re-trained {L}anguage {M}odels",
    author = "Ushio, Asahi  and
      Schockaert, Steven  and
      Camacho-Collados, Jose",
    booktitle = "EMNLP 2021",
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
}

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご