Llama3 TAIDE LX 8B Chat Alpha1

🚀 Llama3 TAIDE Series Models

The TAIDE project is dedicated to developing generative AI dialogue engine models that suit Taiwan's language and cultural characteristics while building a trustworthy AI environment. By integrating industry, academia, and research resources, it promotes the development of trustworthy generative AI, enhances Taiwan's international competitiveness, boosts industrial development, and reduces dependence on foreign technologies.

• The Llama3 TAIDE series of models are based on LLaMA3-8b released by Meta. They incorporate text and training materials applicable to different fields in Taiwan, improving the models' ability to respond in Traditional Chinese and performance in specific tasks. The publicly released models are as follows:
  • Llama3-TAIDE-LX-8B-Chat-Alpha1: Based on LLaMA3-8b, it undergoes continuous pretraining with Traditional Chinese data and instruction tuning to enhance its capabilities in common office tasks and multi-round Q&A conversations. It is suitable for chat and task assistance scenarios. Additionally, a 4-bit quantized model is available. The quantized model is provided for user convenience but may affect performance and cause unexpected issues. Please understand and take note.

🚀 Quick Start

You need to agree to the license terms before using this model. Fill in the following information and click "Submit" to indicate your acceptance of the Llama3-TAIDE Model Community License Agreement and the Privacy Policy.

Click "Submit" to proceed.

✨ Features

• Highly reliable training data: Strictly screen the training data to improve the trustworthiness and applicability of the generated content.
• Enhanced office task performance: Strengthen the model's performance in common office tasks such as automatic summarization, letter writing, article writing, Chinese-English translation, and English-Chinese translation.
• Local knowledge enhancement: Incorporate knowledge of Taiwan's local culture, language usage, and national conditions.
• Multi-round Q&A capabilities: Support multi-round conversations to provide more comprehensive assistance.

💻 Usage Examples

Basic Usage

General Q&A

chat = [
    {"role": "user", "content": "{question}"},
]
prompt = tokenizer.apply_chat_template(chat)

Replace {question} with the user's input.

With System Prompt

chat = [
    {"role": "system", "content": "{sys}"},
    {"role": "user", "content": "{question}"},
]
prompt = tokenizer.apply_chat_template(chat)

Replace {sys} with an instruction, e.g., "You are an AI assistant from Taiwan named TAIDE. You are willing to help users from a Taiwanese perspective and will answer questions in Traditional Chinese." Replace {question} with the user's question.

Multi-round Q&A

chat = [
    {"role": "system", "content": "{sys}"},
    {"role": "user", "content": "{question1}"},
    {"role": "assistant", "content": "{model_anwer_1}"},
    {"role": "user", "content": "{question2}"},
]
prompt = tokenizer.apply_chat_template(chat)

Replace {sys} with an instruction, {question1} with the user's first question, {model_anwer_1} with the model's first answer, and {question2} with the user's second question.

For more details, please refer to Llama3 Documentation.

Advanced Usage

Here are some application examples:

📚 Documentation

Model Parameters

Training Method

Hardware and Software Specifications

• Hardware: H100 from the National Center for High - Performance Computing
• Training Framework: PyTorch

Data Preprocessing

• Character Standardization: Standardize characters to a unified format.
• Duplicate Removal: Eliminate duplicate data.
• Noise Removal:
  • Remove HTML tags and JavaScript code from web data.
  • Eliminate non - standard characters or garbled characters.
  • Filter out articles with too few words.
  • Remove specific formatting in articles, such as line breaks for typesetting.
• Personal Information Removal: Remove personal information such as email addresses and phone numbers.
• Inappropriate Content Removal: Eliminate inappropriate text such as gambling and pornographic content.

Continuous Pretraining (CP)

• Knowledge Enrichment: Supplement a large amount of reliable Traditional Chinese knowledge.
• Hyperparameters:
  • Optimizer: AdamW
  • Learning Rate: 1e - 4
  • Batch Size: 1M tokens
  • Epoch: 1

Fine - Tuning (FT)

• Task - Specific Adaptation: Enable the model to answer questions in Traditional Chinese.
• Hyperparameters:
  • Optimizer: AdamW
  • Learning Rate: 5e - 5
  • Batch Size: 256K tokens
  • Epoch: 3

Training Data

Continuous Pretraining Data (Approximately 140G)

Fine - Tuning Data

The TAIDE team trained the Llama2 series of models to generate fine - tuning data. The generated tasks include single - round or multi - round Q&A on world knowledge, creative writing, general knowledge, translation, summarization, programming, and Taiwan - related values, totaling 128K records. The fine - tuning data will be released later.

Model Evaluation

taide - bench

• Evaluation Data: 500 questions in article writing, letter writing, summarization, English - Chinese translation, and Chinese - English translation. Data link: [taide - bench](https://huggingface.co/datasets/taide/taide - bench).
• Evaluation Method: Scored by GPT4. Scoring program: [taide - bench - eval](https://github.com/taide - taiwan/taide - bench - eval).
• Evaluation Scores | Model | Chinese - English Translation | English - Chinese Translation | Summarization | Article Writing | Letter Writing | Average | | ---- | ---- | ---- | ---- | ---- | ---- | ---- | | Llama3 - TAIDE - LX - 8B - Chat - Alpha1 | 7.770 | 8.280 | 8.495 | 9.605 | 8.950 | 8.620 | | GPT3.5 | 8.880 | 8.810 | 7.450 | 9.490 | 8.750 | 8.676 | | TAIDE - LX - 7B - Chat | 7.165 | 7.685 | 7.720 | 9.635 | 9.110 | 8.263 | | LLAMA2 7B | 6.075 | 4.475 | 5.905 | 2.625 | 3.040 | 4.424 | | LLAMA2 13B | 6.480 | 6.135 | 6.110 | 2.565 | 3.000 | 4.858 | | LLAMA2 70B | 6.975 | 6.375 | 6.795 | 2.625 | 2.990 | 5.152 |

🔧 Technical Details

The model is based on the LLaMA3 - 8b architecture. Through continuous pretraining and fine - tuning with a large amount of Traditional Chinese data, it enhances the model's ability to understand and generate Traditional Chinese text. The data preprocessing steps ensure the quality of the training data, and the carefully selected hyperparameters optimize the training process.

📄 License

• [Llama3 - TAIDE Model Community License Agreement](https://drive.google.com/file/d/12 - Q0WWSjG0DW6CqJQm_jr5wUGRLeb - 8p/view)

⚠️ Important Note

Due to the limitations of the design architecture and potential biases in the data, any responses from the language model do not represent the stance of TAIDE. It is necessary to add additional security protection mechanisms before use, and the response content may also contain incorrect information. Please do not fully trust it.

👥 Development Team

• https://taide.tw/index/teamList

🔗 Related Links

• TAIDE Official Website
• TAIDE Huggingface
• [TAIDE Github](https://github.com/taide - taiwan)
• Kuwa AI

📖 Citation

• TAIDE Official Website