Luke Japanese Base Finetuned Jsts

🚀 Luke Japanese Base Fine-tuned for JSTS

This model is fine-tuned based on luke-japanese-base for JSTS (Sentence Similarity Calculation). It can be used to calculate sentence similarity, with a maximum score of 5.

🚀 Quick Start

Install transformers and sentencepiece, and then execute the following code to solve the JSTS (Sentence Similarity Calculation) task:

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import numpy as np

tokenizer=AutoTokenizer.from_pretrained('Mizuiro-sakura/luke-japanese-base-finetuned-jsts')
model=AutoModelForSequenceClassification.from_pretrained('Mizuiro-sakura/luke-japanese-base-finetuned-jsts')
sentence1='今日は銀座に買い物に出かけた'
sentence2='今日は銀座に服を買いに出かけた'

token=tokenizer(sentence1,sentence2)

import torch
tensor_input_ids = torch.tensor(token["input_ids"])
tensor_attention_masks = torch.tensor(token["attention_mask"])

outputs=model(tensor_input_ids.unsqueeze(0), tensor_attention_masks.unsqueeze(0))

print(outputs.logits[0][1]*5)

✨ Features

• This model is fine-tuned based on luke-japanese-base using the JSTS dataset from Yahoo Japan's JGLUE (https://github.com/yahoojapan/JGLUE).
• It can be used for the task of calculating sentence similarity.

🔧 Technical Details

Model Accuracy

The accuracy of the model is as follows:

• Pearson (Pearson's product-moment correlation coefficient): 0.8971

📄 License

This model is released under the MIT license.

📚 Documentation

What is Luke?

LUKE (Language Understanding with Knowledge-based Embeddings) is a new pre-trained contextualized representation of words and entities based on a transformer. LUKE treats words and entities in a given text as independent tokens and outputs contextualized representations of them. It adopts an entity-aware self-attention mechanism, which is an extension of the self-attention mechanism of the transformer, and considers the types of tokens (words or entities) when computing attention scores.

LUKE achieves state-of-the-art results on five popular NLP benchmarks, including SQuAD v1.1 (extractive question answering), CoNLL-2003 (named entity recognition), ReCoRD (cloze-style question answering), TACRED (relation classification), and Open Entity (entity typing). luke-japanese is the Japanese version of LUKE, a knowledge-enhanced pre-trained Transformer model for words and entities. LUKE treats words and entities as independent tokens and outputs context-aware representations of them.

Acknowledgments

I would like to thank Mr. Yamada (@ikuyamada) and Studio ousia (@StudioOusia), the developers of Luke.

Citation

[1]@inproceedings{yamada2020luke, title={LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention}, author={Ikuya Yamada and Akari Asai and Hiroyuki Shindo and Hideaki Takeda and Yuji Matsumoto}, booktitle={EMNLP}, year={2020} }