Roberta Qa Japanese

🚀 roberta_qa_japanese

This model is designed for extractive question answering in Japanese, leveraging a fine - tuned RoBERTa base model.

This model is a fine - tuned version of [rinna/japanese - roberta - base](https://huggingface.co/rinna/japanese - roberta - base) (a pre - trained RoBERTa model provided by rinna Co., Ltd.) trained specifically for extractive question answering. It is fine - tuned on the JaQuAD dataset provided by Skelter Labs. The data in this dataset is collected from Japanese Wikipedia articles and manually annotated.

✨ Features

• Fine - tuned for Japanese: Specifically optimized for Japanese extractive question - answering tasks.
• Based on RoBERTa: Utilizes the pre - trained RoBERTa architecture for better performance.

🚀 Quick Start

Using a Dedicated Pipeline

from transformers import pipeline

model_name = "tsmatz/roberta_qa_japanese"
qa_pipeline = pipeline(
    "question - answering",
    model=model_name,
    tokenizer=model_name)
result = qa_pipeline(
    question = "決勝トーナメントで日本に勝ったのはどこでしたか。",
    context = "日本は予選リーグで強豪のドイツとスペインに勝って決勝トーナメントに進んだが、クロアチアと対戦して敗れた。",
    align_to_words = False,
)
print(result)

Manual Forward Pass

import torch
import numpy as np
from transformers import AutoModelForQuestionAnswering, AutoTokenizer

model_name = "tsmatz/roberta_qa_japanese"
model = (AutoModelForQuestionAnswering
         .from_pretrained(model_name))
tokenizer = AutoTokenizer.from_pretrained(model_name)

def inference_answer(question, context):
    question = question
    context = context
    test_feature = tokenizer(
        question,
        context,
        max_length=318,
    )
    with torch.no_grad():
        outputs = model(torch.tensor([test_feature["input_ids"]]))
    start_logits = outputs.start_logits.cpu().numpy()
    end_logits = outputs.end_logits.cpu().numpy()
    answer_ids = test_feature["input_ids"][np.argmax(start_logits):np.argmax(end_logits)+1]
    return "".join(tokenizer.batch_decode(answer_ids))

question = "決勝トーナメントで日本に勝ったのはどこでしたか。"
context = "日本は予選リーグで強豪のドイツとスペインに勝って決勝トーナメントに進んだが、クロアチアと対戦して敗れた。"
answer_pred = inference_answer(question, context)
print(answer_pred)

🔧 Technical Details

Training Procedure

You can download the source code for fine - tuning from [here](https://github.com/tsmatz/huggingface - finetune - japanese/blob/master/03 - question - answering.ipynb).

Training Hyperparameters

The following hyperparameters were used during training:

• learning_rate: 7e - 05
• train_batch_size: 2
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon = 1e - 08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 3

Training Results

Framework Versions

• Transformers 4.23.1
• Pytorch 1.12.1+cu102
• Datasets 2.6.1
• Tokenizers 0.13.1

📄 License

This project is licensed under the MIT license.