Kanana Nano 2.1b Embedding

🚀 Kanana

Kanana是由Kakao开发的一系列双语语言模型，在韩语方面表现卓越，在英语方面也具备竞争力。与同规模的先进模型相比，Kanana的计算成本显著降低。

🚀 快速开始

🤗 HuggingFace Transformers

运行Kanana模型需要transformers>=4.45.0或最新版本。

pip install transformers>=4.45.0

kanana-nano-2.1b-embedding的使用示例

⚠️ 重要提示

在使用kanana-nano-2.1b-embedding模型之前，你需要通过pip install datasets安装datasets。

import torch.nn.functional as F
from transformers import AutoModel

instruction = "Given a question, retrieve passages that answer the question"
queries = [
    "are judo throws allowed in wrestling?", 
    "how to become a radiology technician in michigan?",
]

passages = [
    "Since you're reading this, you are probably someone from a judo background or someone who is just wondering how judo techniques can be applied under wrestling rules. So without further ado, let's get to the question. Are Judo throws allowed in wrestling? Yes, judo throws are allowed in freestyle and folkstyle wrestling. You only need to be careful to follow the slam rules when executing judo throws. In wrestling, a slam is lifting and returning an opponent to the mat with unnecessary force.",
    "Below are the basic steps to becoming a radiologic technologist in Michigan:Earn a high school diploma. As with most careers in health care, a high school education is the first step to finding entry-level employment. Taking classes in math and science, such as anatomy, biology, chemistry, physiology, and physics, can help prepare students for their college studies and future careers.Earn an associate degree. Entry-level radiologic positions typically require at least an Associate of Applied Science. Before enrolling in one of these degree programs, students should make sure it has been properly accredited by the Joint Review Committee on Education in Radiologic Technology (JRCERT).Get licensed or certified in the state of Michigan.",
]

model = AutoModel.from_pretrained(
    "kakaocorp/kanana-nano-2.1b-embedding",
    trust_remote_code=True,
).to("cuda")

max_length = 512
query_embeddings = model.encode(queries, instruction=instruction, max_length=max_length)
passage_embeddings = model.encode(passages, instruction="", max_length=max_length)

# get the embeddings with DataLoader (spliting the datasets into multiple mini-batches)
# batch_size = 2
# query_embeddings = model._do_encode(queries, batch_size=batch_size, instruction=instruction, max_length=max_length)
# passage_embeddings = model._do_encode(passages, batch_size=batch_size, instruction="", max_length=max_length)

query_embeddings = F.normalize(query_embeddings, p=2, dim=1)
passage_embeddings = F.normalize(passage_embeddings, p=2, dim=1)


scores = (query_embeddings @ passage_embeddings.T) * 100
print(scores.tolist())

# Output:
# [[84.36527252197266, 31.752296447753906], [35.940425872802734, 81.82719421386719]]

✨ 主要特性

我们推出的Kanana是一系列由Kakao开发的双语语言模型，在韩语方面表现卓越，在英语方面也具备竞争力。与同规模的先进模型相比，Kanana的计算成本显著降低。报告详细介绍了预训练期间采用的技术，以实现计算高效且有竞争力的模型，包括高质量数据过滤、分阶段预训练、深度扩展以及剪枝和蒸馏。此外，报告还概述了Kanana模型后训练期间采用的方法，包括有监督微调以及偏好优化，旨在增强其与用户无缝交互的能力。最后，报告详细说明了用于语言模型适应特定场景的可行方法，如嵌入、函数调用和检索增强生成（RAG）。Kanana模型系列的参数范围从21亿到325亿，其中21亿参数的模型（基础版、指令版、嵌入版、函数调用版和RAG版）已公开发布，以推动韩语语言模型的研究。

⚠️ 重要提示

预训练和后训练数据均不包含Kakao用户数据。

📊 性能表现

以下是Kanana模型系列性能的部分报告。完整结果请参考技术报告。

预训练模型性能

后训练模型性能

指令遵循基准测试

通用基准测试

嵌入模型性能

📚 详细文档

新闻动态

• 📜2025/02/27：发布技术报告和🤗HF模型权重。
• 📕2025/01/10：发布关于Kanana-Nano模型开发的博客文章。(Kanana-Nano)
• 📕2024/11/14：发布关于Kanana模型开发的博客文章。(Kanana LLM: Pre-training，Kanana LLM: Post-training)
• ▶️2024/11/06：发布关于Kanana模型开发的演示视频。(if(kakaoAI)2024)

📄 许可证

Kanana模型采用CC-BY-NC-4.0许可证。

📚 引用信息

@misc{kananallmteam2025kananacomputeefficientbilinguallanguage,
      title={Kanana: Compute-efficient Bilingual Language Models}, 
      author={Kanana LLM Team and Yunju Bak and Hojin Lee and Minho Ryu and Jiyeon Ham and Seungjae Jung and Daniel Wontae Nam and Taegyeong Eo and Donghun Lee and Doohae Jung and Boseop Kim and Nayeon Kim and Jaesun Park and Hyunho Kim and Hyunwoong Ko and Changmin Lee and Kyoung-Woon On and Seulye Baeg and Junrae Cho and Sunghee Jung and Jieun Kang and EungGyun Kim and Eunhwa Kim and Byeongil Ko and Daniel Lee and Minchul Lee and Miok Lee and Shinbok Lee and Gaeun Seo},
      year={2025},
      eprint={2502.18934},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.18934}, 
}

👥 贡献者

• 预训练：Yunju Bak、Doohae Jung、Boseop Kim、Nayeon Kim、Hojin Lee、Jaesun Park、Minho Ryu
• 后训练：Jiyeon Ham、Seungjae Jung、Hyunho Kim、Hyunwoong Ko、Changmin Lee、Daniel Wontae Nam、Kyoung-Woon On
• 适配：Seulye Baeg、Junrae Cho、Taegyeong Eo、Sunghee Jung、Jieun Kang、EungGyun Kim、Eunhwa Kim、Byeongil Ko、Daniel Lee、Donghun Lee、Minchul Lee、Miok Lee、Shinbok Lee、Minho Ryu、Gaeun Seo

📞 联系我们

• Kanana LLM团队技术支持：kanana-llm@kakaocorp.com
• 商务合作联系：alpha.k@kakaocorp.com