Ko Llama3 Luxia 8B

🚀 Ko-Llama3-Luxia-8B

The Ko-Llama3-Luxia-8B model, trained and released by the Language Model Team of Saltlux AI Labs, is a model specialized for the Korean language based on the Llama-3-8B model released by Meta. It offers high - quality text generation capabilities for Korean language tasks.

✨ Features

• Korean Specialization: The Ko-Llama3-Luxia-8B model is specialized for the Korean language, leveraging about 100GB of carefully selected Korean training data from over 1TB of in - house data for pre - training.
• Optimized Tokenizer: The publicly available Llama - 3 Tokenizer has been extended for Korean and used in pre - training, improving tokenization results for Korean text.
• Based on Meta Llama - 3: Built upon the Meta Llama 3 family of large language models, which are optimized for dialogue use cases and perform well on common industry benchmarks.

📚 Documentation

Model Details

The Ko-Llama3-Luxia-8B model, trained and released by the Language Model Team of Saltlux AI Labs, is a model specially tailored for the Korean language based on the Llama - 3 - 8B model released by Meta.

Approximately 100GB of data was selected from over 1TB of in - house Korean training data for pre - training. Additionally, the publicly available Llama - 3 Tokenizer was extended for Korean and used in pre - training.

• Meta Llama - 3: Meta developed and released the Meta Llama 3 family of large language models (LLMs), a collection of pretrained and instruction - tuned generative text models in 8 and 70B sizes. The Llama 3 instruction - tuned models are optimized for dialogue use cases and outperform many of the available open - source chat models on common industry benchmarks. Further, in developing these models, great care was taken to optimize helpfulness and safety.
• License: Llama3 License https://llama.meta.com/llama3/license

Intended Use

The Ko - Llama3 - Luxia - 8B model is developed for research purposes and can be freely trained and utilized for various natural language generation tasks.

Training Details

The pre - training data for Korean specialization was a corpus of about 100GB (as of ~2023), consisting of domains such as news, law, patents, medicine, history, society, culture, and dialogue (both written and spoken) held by Saltlux.

• The currently provided model is trained for 1 Epoch.

Use Device

Pre - training was carried out using 8 NVIDIA H100 80GB devices.

Training Hyperparameters

Tokenizer

To specialize the Llama - 3 - Tokenizer for Korean, 17,536 Korean tokens were added and used.

Tokenizer Result

Korean Input Comparison

English Input Comparison

Inference Result

💻 Usage Examples

Basic Usage

import transformers
import torch

model_id = "saltlux/Ko-Llama3-Luxia-8B"

pipeline = transformers.pipeline(
    "text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto"
)
pipeline("<|begin_of_text|>안녕하세요. 솔트룩스 AI Labs 입니다.")

📄 License

This model is under the Llama3 License. You can find more details here: https://llama.meta.com/llama3/license

📚 Citation instructions

Ko-Llama3-Luxia-8B

@article{kollama3luxiamodelcard,
  title={Ko Llama 3 Luxia Model Card},
  author={AILabs@Saltux},
  year={2024},
  url={https://huggingface.co/saltlux/Ko-Llama3-Luxia-8B/blob/main/README.md}
}

Original Llama-3

@article{llama3modelcard,
title={Llama 3 Model Card},
author={AI@Meta},
year={2024},
url={https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md}
}