Llama-3.2-1B-it-chinese-kyara Open-source Model - Enhance Knowledge Adaptation and Traditional Chinese Comprehension Ability

Llama 3.2 1B It Chinese Kyara

Developed by zake7749

Kyara is an experimental project that improves language models through the knowledge retrieval process, focusing on enhancing the model's knowledge adaptation and language understanding abilities, especially for low-resource languages such as Traditional Chinese.

Large Language Model

Transformers

Supports Multiple Languages#Enhanced Traditional Chinese Support #Fine-tuning for Knowledge Retrieval #Zero-shot Learning

Downloads 1,533

Release Time : 11/20/2024

Model Overview

Kyara aims to improve the performance of large language models on low-resource languages such as Traditional Chinese by expanding their corpora. This project uses knowledge retrieval enhancement technology to address the issue of insufficient training data.

Model Features

Knowledge Retrieval Enhancement

Improve the language model through the knowledge retrieval process and enhance the model's knowledge adaptation ability

Support for Low-resource Languages

Optimize specifically for low-resource languages such as Traditional Chinese

Performance Improvement

Outperform the base model in multiple benchmark tests

Model Capabilities

Text Generation

Knowledge Retrieval

Multilingual Processing

Use Cases

Education

Knowledge Q&A

Used to answer questions in various academic fields

Scored 32.56 in the STEM field of the TMMLUPlus test

Research

Language Model Research

Used to study methods for improving the performance of low-resource language models

🚀 Kyara: Knowledge Yielding Adaptive Retrieval Augmentation for LLM Fine - tuning

Kyara is an experimental project that uses knowledge retrieval to enhance language models, especially for under - represented languages like Traditional Chinese.

🚀 Quick Start

Kyara (Knowledge Yielding Adaptive Retrieval Augmentation) is an experimental project aimed at improving language models through knowledge retrieval processes. The project seeks to enhance the model's ability to adapt knowledge and improve language comprehension, particularly in underrepresented languages like Traditional Chinese. Given the relatively scarce availability of Traditional Chinese data compared to the vast corpus of English data used for model training, Kyara addresses this gap by expanding the limited corpus for this language.

This is a preview model, with the stable version set to be released soon.

✨ Features

Knowledge Retrieval: Improves language models by integrating knowledge retrieval processes.
Language Adaptation: Enhances the model's ability to adapt knowledge and improve language comprehension, especially for under - represented languages such as Traditional Chinese.
Corpus Expansion: Addresses the data scarcity issue in Traditional Chinese by expanding its limited corpus.

📚 Documentation

Benchmark

All evaluations are conducted in a zero - shot setting.

Property	Details
Model Type	Kyara (Knowledge Yielding Adaptive Retrieval Augmentation)
Training Data	Aims to expand the limited corpus of Traditional Chinese data

Metric	Kyara - 1b - it	Llama3.2 - 1b - it
TMMLUPlus	31.92	30.48
- STEM	32.56	29.74
- Humanities	30.60	29.89
- Other	31.08	30.32
- Social - Science	33.42	31.98
MMLU - Redux	41.40	19.62†
GSM8K	31.31	31.61
MATH - L5	5.55	2.91
CRUX	14	11
[AlpacaEval](https://github.com/tatsu - lab/alpaca_eval)	10.79	7.39

†: Llama3.2 - 1b - it appears to have failed to follow the output schema of ZeroEval on MMLU, with 45.28% of examples lacking answers, which has resulted in a lower MMLU score.