Llama 3.2 Korean Bllossom 3B

🚀 Bllossom

Bllossom-3B is an enhanced bilingual model in Korean and English, based on the base model that doesn't support Korean, with additional pre - training on 150GB of refined Korean data and refined instruction tuning.

🚀 Quick Start

Our Bllossom team is releasing the Bllossom - 3B model.
When llama3.2 - 3B came out, it didn't include Korean?? This Bllossom - 3B is an enhanced model in Korean - English based on the base model that doesn't support Korean.
 - It has been additionally pre - trained with 150GB of refined Korean data through 100% full - tuning. (We burned a lot of GPUs)
 - We have carried out highly refined Instruction Tuning.
 - It is a complete bilingual model that doesn't damage English performance at all.
 - We only conducted Instruction tuning. Try tuning it with methods like DPO to improve performance.
 - We didn't use correct answer data or target benchmarks like MT - Bench and LogicKor for training to get good benchmark scores. (If you target these benchmarks for training, you can get a score of 8...)

As always, this model can be used commercially.

1. Bllossom has been presented at AAAI2024, NAACL2024, and LREC - COLING2024 (orally).
2. We will continue to update with good language models!! We welcome anyone who wants to conduct joint research (especially on papers) to enhance the Korean language.

✨ Features

• Language Support: Supports both Korean and English, enhancing the base model's language capabilities.
• Training Method: 100% full - tuning with 150GB of refined Korean data for additional pre - training and refined instruction tuning.
• Bilingual Performance: Maintains English performance while adding Korean support.

📦 Installation

The provided README doesn't have specific installation steps, so this section is skipped.

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = 'Bllossom/llama-3.2-Korean-Bllossom-3B'

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
instruction = "철수가 20개의 연필을 가지고 있었는데 영희가 절반을 가져가고 민수가 남은 5개를 가져갔으면 철수에게 남은 연필의 갯수는 몇개인가요?"

messages = [
    {"role": "user", "content": f"{instruction}"}
    ]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

terminators = [
    tokenizer.convert_tokens_to_ids("<|end_of_text|>"),
    tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

outputs = model.generate(
    input_ids,
    max_new_tokens=1024,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.6,
    top_p=0.9
)

print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True))

Advanced Usage

The provided README doesn't have advanced usage examples, so this part is skipped.

📚 Documentation

Update Information

• [2024.12.06] Updated to the much more powerful latest Bllossom - AICA - 5B Link
• [2024.10.08] The Bllossom - 3B model was first updated.

Model Information

Supported by

• AICA

Citation

Language Model

@misc{bllossom,
  author = {ChangSu Choi, Yongbin Jeong, Seoyoon Park, InHo Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, HyeJin Lee, Younggyun Hahm, Hansaem Kim, KyungTae Lim},
  title = {Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean},
  year = {2024},
  journal = {LREC-COLING 2024},
  paperLink = {\url{https://arxiv.org/pdf/2403.10882}},
 },
}

Vision - Language Model

@misc{bllossom-V,
  author = {Dongjae Shin, Hyunseok Lim, Inho Won, Changsu Choi, Minjun Kim, Seungwoo Song, Hangyeol Yoo, Sangmin Kim, Kyungtae Lim},
  title = {X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment},
  year = {2024},
  publisher = {GitHub},
  journal = {NAACL 2024 findings},
  paperLink = {\url{https://arxiv.org/pdf/2403.11399}},
 },
}

Contact

• KyungTae Lim, Professor at Seoultech. ktlim@seoultech.ac.kr
• Younggyun Hahm, CEO of Teddysum. hahmyg@teddysum.ai
• Hansaem Kim, Professor at Yonsei. khss@yonsei.ac.kr

Contributor

• Hangyeol Yoo, hgyoo@seoultech.ac.kr
• Chansu Choi, choics2623@seoultech.ac.kr