Xlnet Japanese

🚀 XLNet-japanese

XLNet-japanese is a model that utilizes Mecab and sentencepiece with XLNetTokenizer, designed for Japanese language processing.

✨ Features

• This model requires Mecab and sentencepiece with XLNetTokenizer. For more details, refer to this link.
• It uses NFKD as the normalization method for character encoding, which may cause the loss of Japanese muddle marks and semi-muddle marks.

💻 Usage Examples

Basic Usage

from fugashi import Tagger 

from transformers import (
    pipeline,
    XLNetLMHeadModel,
    XLNetTokenizer
)

class XLNet():
    def __init__(self):
        self.m = Tagger('-Owakati') 
        self.gen_model = XLNetLMHeadModel.from_pretrained("hajime9652/xlnet-japanese")
        self.gen_tokenizer = XLNetTokenizer.from_pretrained("hajime9652/xlnet-japanese")
         
    def generate(self, prompt="福岡のご飯は美味しい。コンパクトで暮らしやすい街。"):
        prompt = self.m.parse(prompt)
        inputs = self.gen_tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")
        prompt_length = len(self.gen_tokenizer.decode(inputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True))
        outputs = self.gen_model.generate(inputs, max_length=200, do_sample=True, top_p=0.95, top_k=60)
        generated = prompt + self.gen_tokenizer.decode(outputs[0])[prompt_length:]
        return generated

🔧 Technical Details

• This model's training uses the Japanese Business News dataset.
• The normalization method for character encoding is NFKD, which may lead to the loss of Japanese muddle marks and semi-muddle marks.

📄 License

This model is licensed under the Apache-2.0 license.

⚠️ Important Note

The company that created and published this model is called Stockmark. This repository is for use by HuggingFace and not for infringement. See this document published by https://github.com/mkt3.