Llama 2 Ko 7b

🚀 Llama-2-Ko 🦙🇰🇷

Llama-2-Ko is an advanced version of Llama 2. It benefits from an expanded vocabulary and includes a Korean corpus in further pretraining. Similar to its predecessor, it's part of generative text models with parameter counts ranging from 7 billion to 70 billion. This repository focuses on the 7B pretrained version in the Hugging Face Transformers format. For other models, refer to the index below.

🚀 Quick Start

There is no specific quick - start content in the original document, so this section is skipped.

✨ Features

Update Log

• 2023.12.27: A new model is available! It's trained with only open - accessible Korean text corpus: https://huggingface.co/beomi/open-llama-2-ko-7b
• 2023.10.19: Fixed the tokenizer bug (space not applied when decoding) after transforemrs>=4.34.0

📚 Documentation

Model Details

• Model Developers: Junbum Lee (Beomi)
• Variations: Llama-2-Ko will come in parameter sizes of 7B, 13B, and 70B, with pretrained and fine - tuned variations.
• Input: The models take only text as input.
• Output: The models generate only text.
• Model Architecture: Llama-2-Ko is an auto - regressive language model using an optimized transformer architecture based on Llama-2.

Vocab Expansion

Tokenizing "안녕하세요, 오늘은 날씨가 좋네요."

Tokenizing "Llama 2: Open Foundation and Fine - Tuned Chat Models"

Model Benchmark

LM Eval Harness - Korean (polyglot branch)

• Used EleutherAI's lm - evaluation - harness https://github.com/EleutherAI/lm-evaluation-harness/tree/polyglot

NSMC (Acc) - 50000 full test

TBD

COPA (F1)

HellaSwag (F1)

BoolQ (F1)

SentiNeg (F1)

Note for oobabooga/text - generation - webui

Remove ValueError at load_tokenizer function (line 109 or near), in modules/models.py.

diff --git a/modules/models.py b/modules/models.py
index 232d5fa..de5b7a0 100644
--- a/modules/models.py
+++ b/modules/models.py
@@ -106,7 +106,7 @@ def load_tokenizer(model_name, model):
                 trust_remote_code=shared.args.trust_remote_code,
                 use_fast=False
             )
-        except ValueError:
+        except:
             tokenizer = AutoTokenizer.from_pretrained(
                 path_to_model,
                 trust_remote_code=shared.args.trust_remote_code,

Since Llama - 2 - Ko uses FastTokenizer provided by HF tokenizers NOT sentencepiece package, it is required to use use_fast=True option when initialize tokenizer.

Apple Sillicon does not support BF16 computing, use CPU instead. (BF16 is supported when using NVIDIA GPU)

Citation

@misc {l._junbum_2023,
	author       = { {L. Junbum} },
	title        = { llama-2-ko-7b (Revision 4a9993e) },
	year         = 2023,
	url          = { https://huggingface.co/beomi/llama-2-ko-7b },
	doi          = { 10.57967/hf/1098 },
	publisher    = { Hugging Face }
}

Acknowledgement

The training is supported by TPU Research Cloud program.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here