Luke Japanese Wordpiece Base

🚀 Luke Japanese WordPiece Base Model

This is a model based on studio-ousia/luke-japanese-base with the following modifications, designed for named entity recognition tasks.

📦 Model Information

✨ Features

This model has made the following improvements compared to the base model:

• Switched the base model from RoBERTa to Japanese BERT, and accordingly, the tokenizer changed from Sentencepiece to WordPiece.
• Performed pre - training on Japanese Wikipedia data as of July 1, 2023.
• Enabled handling of [UNK] (unknown) entities.

For more details, please refer to the blog post.

🚀 Quick Start

Here is an example of using this model:

from transformers import AutoTokenizer, AutoModel

# Since the code for the tokenizer of this model is used, it is necessary to specify trust_remote_code=True
tokenizer = AutoTokenizer.from_pretrained("uzabase/luke-japanese-wordpiece-base", trust_remote_code=True)

model = AutoModel.from_pretrained("uzabase/luke-japanese-wordpiece-base")

📚 Documentation

Update Information

• November 28, 2023: The following updates were made:
  • Fixed the issue where the tokenizer could not be loaded in transformers v4.34.0 and later.
  • Changed the tokenizer output to include position_ids.
    • Previously, the position_ids automatically assigned by the LUKE model here were used. However, these were in the RoBERTa specification and were not correct for this model using BERT. Therefore, position_ids are now explicitly included in the tokenizer output to ensure the correct values for BERT are input to the model.
  • Removed the "None:" string at the beginning of each token (excluding special tokens such as "[PAD]") in the tokenizer's entity_vocab.
    • For example, the token "None:聖徳太子" has been corrected to "聖徳太子".
• September 7, 2023: The model was released.