Mamba Ko 2.8b

🚀 Mamba-ko-2.8B🐍

Mamba-ko-2.8B is a state space model that has been further pretrained (or continuously trained) using a synthetically generated dataset - korean_textbooks. It aims to leverage the power of state space models for text generation tasks.

✨ Features

• Advanced Architecture: Based on the Mamba state space model architecture, which shows promising performance on information - dense data such as language modeling.
• Korean - Focused: Further pretrained with Korean textbooks dataset, making it more suitable for Korean language tasks.

📦 Installation

pip install causal_conv1d>=1.1.0 mamba-ssm==1.1.1

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer
from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel

device = "cuda" if torch.cuda.is_available() else "cpu"

model_name = "kuotient/mamba-ko-2.8b"
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

model = MambaLMHeadModel.from_pretrained(
        model_name, device=device, dtype=torch.float16)

prompt = "아이들한테 제공할 영양가 있는 음식 5가지의 예시는 다음과 같다."

tokens = tokenizer(prompt, return_tensors='pt')
input_ids = tokens.input_ids.to(device)
streamer = TextStreamer(tokenizer)

out = model.generate(
    input_ids=input_ids,
    streamer=streamer,
    max_length=2000,
    temperature=0.7,
    top_p=0.7,
    eos_token_id=tokenizer.eos_token_id,
)

📚 Documentation

What is Mamba?

Mamba is a new state space model architecture showing promising performance on information - dense data such as language modeling, where previous subquadratic models fall short of Transformers. It is based on the line of progress on structured state space models, with an efficient hardware - aware design and implementation in the spirit of FlashAttention.

TODO

• 🟢 Training with korean_textbooks dataset - DONE
• More training with publicly available Korean corpora
• 🟡 Instruct tuning

Model Details

Model Benchmark

KoBEST

Thanks

We would like to thank maywell for his great contributions and motivation to the Korean LLM community.

📄 License

This project is licensed under the Apache 2.0 license.

💡 Usage Tip If you're interested in building large - scale language models to solve a wide variety of problems in a wide variety of domains, you should consider joining Allganize. For a coffee chat or if you have any questions, please do not hesitate to contact us at kuotient.dev@gmail.com.

The author also thanks Allganize Korea for their generosity in providing resources for this personal project. Note that this project is not directly related to the company's goals or research.