Kanana Nano 2.1b Embedding

🚀 Kanana

Kanana是由Kakao開發的一系列雙語語言模型，在韓語方面表現卓越，在英語方面也具備競爭力。與同規模的先進模型相比，Kanana的計算成本顯著降低。

🚀 快速開始

🤗 HuggingFace Transformers

運行Kanana模型需要transformers>=4.45.0或最新版本。

pip install transformers>=4.45.0

kanana-nano-2.1b-embedding的使用示例

⚠️ 重要提示

在使用kanana-nano-2.1b-embedding模型之前，你需要通過pip install datasets安裝datasets。

import torch.nn.functional as F
from transformers import AutoModel

instruction = "Given a question, retrieve passages that answer the question"
queries = [
    "are judo throws allowed in wrestling?", 
    "how to become a radiology technician in michigan?",
]

passages = [
    "Since you're reading this, you are probably someone from a judo background or someone who is just wondering how judo techniques can be applied under wrestling rules. So without further ado, let's get to the question. Are Judo throws allowed in wrestling? Yes, judo throws are allowed in freestyle and folkstyle wrestling. You only need to be careful to follow the slam rules when executing judo throws. In wrestling, a slam is lifting and returning an opponent to the mat with unnecessary force.",
    "Below are the basic steps to becoming a radiologic technologist in Michigan:Earn a high school diploma. As with most careers in health care, a high school education is the first step to finding entry-level employment. Taking classes in math and science, such as anatomy, biology, chemistry, physiology, and physics, can help prepare students for their college studies and future careers.Earn an associate degree. Entry-level radiologic positions typically require at least an Associate of Applied Science. Before enrolling in one of these degree programs, students should make sure it has been properly accredited by the Joint Review Committee on Education in Radiologic Technology (JRCERT).Get licensed or certified in the state of Michigan.",
]

model = AutoModel.from_pretrained(
    "kakaocorp/kanana-nano-2.1b-embedding",
    trust_remote_code=True,
).to("cuda")

max_length = 512
query_embeddings = model.encode(queries, instruction=instruction, max_length=max_length)
passage_embeddings = model.encode(passages, instruction="", max_length=max_length)

# get the embeddings with DataLoader (spliting the datasets into multiple mini-batches)
# batch_size = 2
# query_embeddings = model._do_encode(queries, batch_size=batch_size, instruction=instruction, max_length=max_length)
# passage_embeddings = model._do_encode(passages, batch_size=batch_size, instruction="", max_length=max_length)

query_embeddings = F.normalize(query_embeddings, p=2, dim=1)
passage_embeddings = F.normalize(passage_embeddings, p=2, dim=1)


scores = (query_embeddings @ passage_embeddings.T) * 100
print(scores.tolist())

# Output:
# [[84.36527252197266, 31.752296447753906], [35.940425872802734, 81.82719421386719]]

✨ 主要特性

我們推出的Kanana是一系列由Kakao開發的雙語語言模型，在韓語方面表現卓越，在英語方面也具備競爭力。與同規模的先進模型相比，Kanana的計算成本顯著降低。報告詳細介紹了預訓練期間採用的技術，以實現計算高效且有競爭力的模型，包括高質量數據過濾、分階段預訓練、深度擴展以及剪枝和蒸餾。此外，報告還概述了Kanana模型後訓練期間採用的方法，包括有監督微調以及偏好優化，旨在增強其與用戶無縫交互的能力。最後，報告詳細說明了用於語言模型適應特定場景的可行方法，如嵌入、函數調用和檢索增強生成（RAG）。Kanana模型系列的參數範圍從21億到325億，其中21億參數的模型（基礎版、指令版、嵌入版、函數調用版和RAG版）已公開發布，以推動韓語語言模型的研究。

⚠️ 重要提示

預訓練和後訓練數據均不包含Kakao用戶數據。

📊 性能表現

以下是Kanana模型系列性能的部分報告。完整結果請參考技術報告。

預訓練模型性能

後訓練模型性能

指令遵循基準測試

通用基準測試

嵌入模型性能

📚 詳細文檔

新聞動態

• 📜2025/02/27：發佈技術報告和🤗HF模型權重。
• 📕2025/01/10：發佈關於Kanana-Nano模型開發的博客文章。(Kanana-Nano)
• 📕2024/11/14：發佈關於Kanana模型開發的博客文章。(Kanana LLM: Pre-training，Kanana LLM: Post-training)
• ▶️2024/11/06：發佈關於Kanana模型開發的演示視頻。(if(kakaoAI)2024)

📄 許可證

Kanana模型採用CC-BY-NC-4.0許可證。

📚 引用信息

@misc{kananallmteam2025kananacomputeefficientbilinguallanguage,
      title={Kanana: Compute-efficient Bilingual Language Models}, 
      author={Kanana LLM Team and Yunju Bak and Hojin Lee and Minho Ryu and Jiyeon Ham and Seungjae Jung and Daniel Wontae Nam and Taegyeong Eo and Donghun Lee and Doohae Jung and Boseop Kim and Nayeon Kim and Jaesun Park and Hyunho Kim and Hyunwoong Ko and Changmin Lee and Kyoung-Woon On and Seulye Baeg and Junrae Cho and Sunghee Jung and Jieun Kang and EungGyun Kim and Eunhwa Kim and Byeongil Ko and Daniel Lee and Minchul Lee and Miok Lee and Shinbok Lee and Gaeun Seo},
      year={2025},
      eprint={2502.18934},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.18934}, 
}

👥 貢獻者

• 預訓練：Yunju Bak、Doohae Jung、Boseop Kim、Nayeon Kim、Hojin Lee、Jaesun Park、Minho Ryu
• 後訓練：Jiyeon Ham、Seungjae Jung、Hyunho Kim、Hyunwoong Ko、Changmin Lee、Daniel Wontae Nam、Kyoung-Woon On
• 適配：Seulye Baeg、Junrae Cho、Taegyeong Eo、Sunghee Jung、Jieun Kang、EungGyun Kim、Eunhwa Kim、Byeongil Ko、Daniel Lee、Donghun Lee、Minchul Lee、Miok Lee、Shinbok Lee、Minho Ryu、Gaeun Seo

📞 聯繫我們

• Kanana LLM團隊技術支持：kanana-llm@kakaocorp.com
• 商務合作聯繫：alpha.k@kakaocorp.com