Gemma 2 2b It Chinese Kyara Dpo

🚀 Kyara: Knowledge Yielding Adaptive Retrieval Augmentation for LLM Fine-tuning

Kyara is an experimental project that aims to enhance language models through knowledge retrieval. It improves the model's knowledge adaptation and language comprehension, especially for underrepresented languages like Traditional Chinese.

🤗 Hugging Face  ｜ 🚀Github  ｜  📑 Paper  ｜  📖 English  ｜  📖 Chinese  ｜  💻 Kaggle Notebook

🚀 Quick Start

Kyara (Knowledge Yielding Adaptive Retrieval Augmentation) is an experimental project aimed at improving language models through knowledge retrieval processes. The project seeks to enhance the model’s ability to adapt knowledge and improve language comprehension, particularly in underrepresented languages like Traditional Chinese. Given the relatively scarce availability of Traditional Chinese data compared to the vast corpus of English data used for model training, Kyara addresses this gap by expanding the limited corpus for this language.

To validate Kyara's effectiveness, we conducted full-parameter fine-tuning on Gemma-2-2b-it, resulting in the first iteration of the Kyara model. Initial evaluation results, as detailed in the Benchmark section, demonstrate that Kyara outperforms the original Gemma-2-2b-it across various benchmarks, with notable improvements in Chinese language evaluations.

✨ Features

Retrieval Augmented Generation (Experimental)

Benefiting from Kyara's training method, we incorporated RAG-related content during the SFT phase. You can refer to the following examples to construct task templates.

📚 Documentation

Benchmark

General Benchmark

The following evaluations are based on zero-shot.

The aggregation method for the groups in TMMLUPlus is macro average, following the practice in the official implementation.

Open-LLM Leaderboard

As of now, Kyara-2b-it is the leading competitor among all 2b-scale models on the OpenLLM Leaderboard.

Alignment Benchmark

AlignBench

where the postfixes CHT and CHS represent Traditional Chinese and Simplified Chinese, respectively. To evaluate the performance on Traditional Chinese in AlignBench, we used OpenCC with the s2tw configuration to convert all questions from Simplified Chinese to Traditional Chinese.

Usage

Kyara adopts the same architecture as Gemma2, utilizing identical inference and training methods. We have created a Jupyter Notebook on Kaggle to demonstrate Kyara’s basic functionality. For service-level deployment, we recommend using Sglang or vllm to achieve greater throughput and robustness.

Method

Dataset Summary

We have collected a total of 3.6M conversations, approximately 4.51 billion tokens. The following provides an overview of the language distribution and conversation rounds.

• Language:

• Conversation Rounds:

Dataset Construction

The data construction for Kyara is divided into two parts: English and Chinese. For the English part, we have incorporated multiple high-quality open-source datasets, such as teknium/OpenHermes-2.5 and arcee-ai/The-Tome, and performing semantic deduplication to drop out near-similar examples. As for the Chinese part, the construction follows the process outlined below.

Base Dataset: Knowledge Injection with Retrieval Augmentation

We developed a knowledge search system using open Chinese knowledge corpora, integrated with QDrant. To construct Supervised Fine-Tuning(SFT) pairs, we followed this process:

1. Sample documents from the knowledge base and generate knowledge-intensive questions that users might ask based on these texts.
2. (Optional) Increase instruction complexity using Evol-Instruct.
3. Apply query expansion on the generated instructions to retrieve additional Top K documents and individually assess their relevance:
   • For relevant documents, use an LLM to summarize key information related to the question.
   • For irrelevant documents, ignore them.
4. Let the LLM generate a detailed and comprehensive response according to the original document and K supplementary references.

Besides, we would also ask the LLM to generate a user prompt for high-quality documents, and pair the (generated prompt, original document) as an SFT example.

Chinese Math Dataset

• Dataset: zake7749/kyara-chinese-math-sft-s0-30K

While the aforementioned strategy can generate a wide range of knowledge-based texts, it primarily falls within the scope of information-seeking tasks and is not very effective in constructing mathematical and reasoning-related content. To address this, we generated 50,000 math problems based on PersonaHub. We then used Gemini-1.5-Flash to filter out data with obvious errors in calculation and reasoning, thereby creating kyara-chinese-math-sft-s0-30K.

High Quality Dataset: Model Refinement

After completing supervised learning using the base dataset, we will fine-tune the LLM again on a high-quality subset, primarily to address the following three issues:

1. Some responses in the Base Dataset were generated from small models, which sometimes performed poorly in following instructions.
2. We used various LLMs in the previous step to introduce knowledge diversity and language adaptability. However, we discovered subtle differences in response templates and reasoning approaches between different LLMs, leading to occasional instability in the trained Chat Model. Therefore, we would like to introduce a high-quality small dataset, using a single strong LLM to generate QA Pairs.
3. The Base Dataset includes some Q&A Pairs composed of generated queries and original documents. While these data are rich in knowledge, they are relatively weak in terms of instruction following.

To balance data diversity and quality, we adopted a strategy similar to InsTag to classify the data. We then used ArmoRM and an LLM Judge to evaluate data quality, finally extracting the best training data from each category to create the Stage 1 Dataset of about 500K, which was used to fine-tune the Kyara-SFT Model again.

Preference Learning

We introduced Preference Learning in Kyara, which allows the model's responses to better align with human preferences while enhancing programming skills and mathematical reasoning abilities.

Kyara’s preference learning strategy utilizes Direct Preference Optimization (DPO), integrating two custom-built Chinese datasets alongside two English datasets.

• argilla/ultrafeedback-binarized-preferences
• xinlai/Math-Step-DPO-10K

Here, we summarize the construction strategy of the Chinese datasets.

Chinese DPO

SPIN/SPPO

We followed the original design, using Kyara-SFT to generate a set of contrastive data for the High Quality Dataset.

RLAIF

Dataset: zake7749/kyara-chinese-preference-dpo-s0-30K

We extracted Chinese Prompts from Magpie-Align/Magpie-Qwen2-Pro-200K-Chinese, hfl/stem_zh_instruction, and FreedomIntelligence/Evol-Instruct-Chinese-GPT4, and distributed the same prompt to four different LLMs. The competitors include:

• GPT-4o
• GPT-4-0618
• ChatGPT-3.5-0513
• Claude-Sonnet-3.5
• Yi-Large
• Mixtral 8x22B
• Gemini-Flash
• Qwen2-72B-Instruct
• DeepSeek V2

After response generation, we ask the LLMs to judge which one is better, using the following prompt:

**[Task]**

Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. Your evaluation should consider correctness and helpfulness.  
1. First, independently solve the user question step-by-step.  
2. Then, compare both assistants’ answers with your answer. Identify and correct any mistakes.  
3. Do not allow the length of the responses to influence your evaluation.  
4. Be as objective as possible.

After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better, and "[[C]]" for a tie or if both A and B are bad.  

If the answers from A and B are very similar in terms of correctness, helpfulness, and relevance, meaning there is no "obvious" winner, judge it as a tie and output [[C]].

**[User Question]**  
{prompt}

---

**[Assistant A’s Answer]**  
{answer}

---

**[Assistant B’s Answer]**  
{prediction}

---

Finally, all four datasets were combined for DPO training.

💻 Usage Examples

Basic Usage

The input example for constructing task templates:

# Reference Document
<reference>
<document>
Document ID: id_27025b13
* Document Title: Flash_memory
* Document Text:
Another limitation of flash memory is its limited number of erase cycles (most commercial SLC flash memory guarantees around 100,000 erase cycles for the "0" zone, but due to manufacturing precision, other blocks are not guaranteed, and some might even have factory defects and be unusable). This limitation is partly offset by firmware or file systems that calculate write counts and perform dynamic remapping to distribute writes across different blocks; this technique is called wear leveling. Another method is known as Bad Block Management (BBM), where blocks are dynamically tested during write operations, and failed blocks are discarded. For most mobile devices, these wear management techniques can extend the life of internal flash memory (sometimes even beyond the device's lifespan). Moreover, partial data loss in these devices may be acceptable. However, for high-reliability data storage applications that require heavy data write cycles, flash memory is not recommended. But this limitation does not apply to read-only applications, such as routers and thin clients, which often only write once or a few times throughout their lifespan.

### Read Disturbance
</document>
<document>
Document ID: id_858b1787
* Document Title: Flash_memory
* Document Text:
* TLC NAND flash memory typically has an endurance of around 1,000 or more cycles (Samsung 840); using multi-layer structures and adopting LDPC correction have extended the endurance.
* QLC NAND flash memory can have an endurance ranging from 500 to 1,000 cycles.
* SLC floating-gate NOR flash memory typically has a write endurance of 100,000 to 1,000,000 cycles (Numonyx M58BW 100k; Spansion S29CD016J 1,000k).
* MLC floating-gate NOR flash memory usually has a write endurance of 100,000 cycles (Numonyx J3 flash).

These values are approximate and depend on the technology and positioning of different manufacturers' products. Finer process technologies can improve read/write performance and capacity, but they may also pose greater challenges in terms of write endurance. Specific algorithms and design examples, such as wear leveling and memory over-provisioning, can be used to adjust storage system endurance to meet specific needs. Wear leveling is essential for ensuring the lifespan of flash memory products, and it is supported in products like USB flash drives and SSDs.

## Flash Memory File Systems
</document>
<document>
Document ID: id_df34eb65
* Document Title: Memory_over-provisioning
* Document Text:
## Basic SSD Operations

Due to the nature of flash memory operations, data cannot be overwritten directly like in hard drives. When data is first written to an SSD, the cells are in an erased state, so the data can be written directly, one page at a time (usually 4 to 8 KB in size). The SSD controller, which manages the flash memory and interfaces with the main control system, uses a logical-to-physical mapping system called Logical Block Addressing (LBA), part of the flash translation layer (FTL). When new data needs to replace old data, the SSD controller writes the new data to a new location and updates the logical mapping to point to the new physical location. The original data becomes invalid and must be erased before it can be rewritten.

Flash memory has a limited number of program/erase (P/E) cycles. Typically, this is expressed as the maximum number of P/E cycles that flash memory can endure over its lifetime. Single-level cell (SLC) flash memory is generally designed for high performance and long life, typically supporting 50,000 to 100,000 cycles. As of 2011, multi-level cell (MLC) flash memory, designed for low-cost applications, has far fewer cycles, usually only 3,000 to 5,000 cycles. Since 2013, triple-level cell (TLC) flash memory has been introduced, with P/E cycles dropping to around 1,000. The lower the write amplification, the better, as it corresponds to fewer P/E cycles, which extends the lifespan of the SSD.
</document>
</reference>

---

# Task Description
Please refer to the content in the <reference> above and answer the user's quest

🔧 Technical Details

Model Information

📄 License

The license for this project is Gemma.