OPEN SOLAR KO 10.7B

🚀 Open-Solar-Ko

Solar-Ko is an advanced version of the upstage/SOLAR-10.7B-v1.0 model. It features an expanded vocabulary and incorporates a Korean corpus for enhanced pretraining. Open-Solar-Ko only uses publicly accessible Korean corpora, such as AI Hub, Modu Corpus, 모두의 말뭉치, and Korean Wikipedia. Since it was trained solely with publicly available corpora, this model can be used freely by everyone under the Apache2.0 open - source license.

🚀 Quick Start

This section is not provided in the original README, so it is skipped.

✨ Features

• Advanced Iteration: Solar-Ko is an advanced version of the upstage/SOLAR-10.7B-v1.0 model.
• Vocab Expansion: It has an expanded vocabulary, which improves tokenizing efficiency for Korean text.
• Open - Source and Public Data: Trained with publicly available Korean corpora, allowing unrestricted use under the Apache2.0 license.

📦 Installation

This section is not provided in the original README, so it is skipped.

💻 Usage Examples

This section is not provided in the original README, so it is skipped.

📚 Documentation

Update Log

• 2024.01.08: Initial Test version Release of Solar-Ko

Model Details

• Model Developers: Junbum Lee (Beomi)
• Variations: Solar-Ko is available with one parameter size — 10B with Continual Pretrained version.
• Input: The model accepts only text input.
• Output: The model produces text output exclusively.
• Model Architecture: SOLAR-KO-10.7B is an auto-regressive language model that leverages an optimized transformer architecture derived from Llama-2.

Training Corpus

The model was trained using selected datasets from AIHub and Modu Corpus. Detailed information about the training datasets is as follows:

• AI Hub: corpus/AI_HUB
  • Only the Training segment of the data was used.
  • The Validation and Test segments were deliberately excluded.
• Modu Corpus: corpus/MODU_CORPUS

The final JSONL dataset used to train this model is approximately 61GB in size. Total token count: Approximately 15 billion tokens (*using the expanded tokenizer. With the original SOLAR tokenizer, >60 billion tokens.)

Vocab Expansion

Tokenizing "안녕하세요, 오늘은 날씨가 좋네요."

• SOLAR-10.7B: 26 tokens
• SOLAR-KO-10.7b: 8 tokens

Tokenizing "Meet 10.7B Solar: Elevating Performance with Upstage Depth UP Scaling!"

• SOLAR-10.7B: 22 tokens
• SOLAR-KO-10.7b: 22 tokens

Model Benchmark

LM Eval Harness - Korean (polyglot branch)

• Used EleutherAI's lm - evaluation - harness https://github.com/EleutherAI/lm - evaluation - harness/tree/polyglot

Citation

@misc {solar_ko_junbum_2023,
    author       = { {L. Junbum} },
    title        = { Solar-Ko-10.7b },
    year         = 2024,
    url          = { https://huggingface.co/beomi/SOLAR-KO-10.7B },
    publisher    = { Hugging Face }
}

Acknowledgements

• Training support was provided by the TPU Research Cloud program.
• The training corpus includes data from AI Hub, Modu Corpus, and Korean Wikipedia.

📄 License

Apache 2.0