Watashiha Gpt 6b

🚀 Comedy Prompt Language Model

This is a comedy prompt language model developed using AWS's trn1 instances. It has undergone pre - training and then fine - tuning with comedy prompt data.

🚀 Quick Start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "watashiha/watashiha-gpt-6b"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)

if torch.cuda.is_available():
    model = model.to("cuda")

text = "お題:ホラー映画の「○○○から逃げろ！」<SEP>回答:"
token_ids = tokenizer.encode(text, add_special_tokens=False, return_tensors="pt").to(model.device)

output_ids = model.generate(
    token_ids,
    do_sample=True,
    max_new_tokens=32,
    top_p=0.9,
    top_k=50,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)
output = tokenizer.decode(output_ids.tolist()[0], skip_special_tokens=True)
print(output)
"""お題:ホラー映画の「○○○から逃げろ！」<SEP>回答:怖いもの知らずの大学生"""

✨ Features

• Model Architecture: Based on the GPT2 architecture.
• Vocabulary Size: 44,880.
• Model Size: 6B parameters.
• License: Apache License 2.0.
• Library: [aws - neuron - reference - for - megatron - lm](https://github.com/aws - neuron/aws - neuron - reference - for - megatron - lm).

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "watashiha/watashiha-gpt-6b"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)

if torch.cuda.is_available():
    model = model.to("cuda")

text = "お題:ホラー映画の「○○○から逃げろ！」<SEP>回答:"
token_ids = tokenizer.encode(text, add_special_tokens=False, return_tensors="pt").to(model.device)

output_ids = model.generate(
    token_ids,
    do_sample=True,
    max_new_tokens=32,
    top_p=0.9,
    top_k=50,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)
output = tokenizer.decode(output_ids.tolist()[0], skip_special_tokens=True)
print(output)
"""お題:ホラー映画の「○○○から逃げろ！」<SEP>回答:怖いもの知らずの大学生"""

Advanced Usage

There is no advanced usage example in the original document, so this part is skipped.

📚 Documentation

Training Data

The pre - training was conducted using the following corpora, with a total of 47.7 billion tokens:

• Japanese data from C4.
• Japanese data from CC - 100.
• Japanese data from OSCAR.
• Japanese dump data from Wikipedia.
• In - house data.

Fine - tuning was performed using 6.93 million comedy prompt data.

Performance Comparison

The following is the result of fine - tuning each model under the same conditions and having the generated jokes evaluated on a four - point scale by a mobile comedy prompt legend:

Out of range: The model fails to understand the topic as Japanese. One star: The model understands the topic but the joke is not well - formed (not funny). Two stars: The joke is well - formed (funny). Three stars: The joke is very funny (above a certain level of funniness).

🔧 Technical Details

There is no specific technical implementation details in the original document, so this section is skipped.

📄 License

This project is licensed under the Apache License 2.0.

👨‍💻 Developers

• UCHIDA, Tatsuya
• KOBASHI, Yohei
• KUROKI, Shuya
• KUBOTA, Hikaru
• TAKENOUCHI, Daisuke