Sarashina2 7b

🚀 Sarashina2-7B

This repository offers large language models trained by SB Intuitions, which can be applied in various natural language processing tasks.

🚀 Quick Start

The following shows how to use the Sarashina2-7B model to generate text.

Basic Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline, set_seed
 
model = AutoModelForCausalLM.from_pretrained("sbintuitions/sarashina2-7b", torch_dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina2-7b")
# If you want to use slow tokenizer
# tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina2-7b", use_fast=False）
generator = pipeline("text-generation", model=model, tokenizer=tokenizer)
set_seed(123)
 
text = generator(
    "おはようございます、今日の天気は",
    max_length=30,
    do_sample=True,
    pad_token_id=tokenizer.pad_token_id,
    num_return_sequences=3,
)

for t in text:
  print(t)
 
# These examples are generated by sarashina2-7b parameters model
# {'generated_text': 'おはようございます、今日の天気は晴れです。ちょっと風が強い。\n昨日は、久しぶりにゆっくりとしていました。\n2週間位間があいてしまったかも、でもその間に'}
# {'generated_text': 'おはようございます、今日の天気は曇。朝は曇っていてどんよりしていましたね。昼からは晴れそうですが。気温は徐々に上昇しています。昨日は春らしい陽気でした。'}
# {'generated_text': 'おはようございます、今日の天気はくもり、少し寒気がします。 この土日に、家族で一泊二日で旅行に行ってきました。といっても、100キロ'}

📚 Documentation

Configuration

Training Corpus

For Japanese training data, the Japanese portion of the Common Crawl corpus, the largest Web corpus, was used as the training dataset. To clean the training corpus, CCNet and HojiChar were employed. After cleaning, the Japanese training data contains about 1T tokens.

For English training data, English documents were extracted from SlimPajama, but the books3 corpus was removed due to copyright infringement.

Tokenization

A sentencepiece tokenizer with a unigram language model and byte-fallback is used. Pre-tokenization with a Japanese tokenizer is not applied. Thus, users can directly input raw sentences into the tokenizer.

Ethical Considerations and Limitations

Sarashina2 has not been tuned to follow instructions yet. Therefore, it might generate some meaningless sequences, inaccurate instances, or biased/objectionable outputs. Before using Sarashina2, developers are advised to tune the models based on human preferences and safety considerations.

📄 License

This project is licensed under the MIT License.