Stockmark 13b

🚀 stockmark/stockmark-13b

Stockmark-13b is a large language model (LLM) with 13 billion parameters. It is pretrained from scratch using a Japanese corpus of approximately 220 billion tokens. This model is developed by Stockmark Inc..

For more detailed information, please refer to our blog.

This project is supported by the AWS LLM development support program.

We also offer stockmark-13b-instruct, which is an instruction-tuned version of stockmark-13b.

🚀 Quick Start

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# For A100 or H100 GPU
model = AutoModelForCausalLM.from_pretrained("stockmark/stockmark-13b", device_map="auto", torch_dtype=torch.bfloat16)

# If you use a T4 or V100 GPU, please load a model in 8 bit with the below code.
# To do so, you need to install `bitsandbytes` via `pip install bitsandbytes`.
# model = AutoModelForCausalLM.from_pretrained("stockmark/stockmark-13b", device_map={"": 0}, load_in_8bit=True)

tokenizer = AutoTokenizer.from_pretrained("stockmark/stockmark-13b")

inputs = tokenizer("自然言語処理とは", return_tensors="pt").to(model.device)
with torch.no_grad():
    tokens = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=0.7
    )
    
output = tokenizer.decode(tokens[0], skip_special_tokens=True)
print(output)

Examples

• LoRA tuning: https://huggingface.co/stockmark/stockmark-13b/blob/main/notebooks/LoRA.ipynb

📚 Documentation

Training dataset

We have used a Japanese corpus with a total of approximately 220 billion tokens.

Accelerator and Library

• Accelerator: AWS Trainium
  • https://aws.amazon.com/machine-learning/trainium/
• Library for distributed training: neuronx-nemo-megatron
  • https://github.com/aws-neuron/neuronx-nemo-megatron

📄 License

MIT

Developed by

Stockmark Inc.

Author

Takahiro Omi