TAIDE LX 7B

🚀 TAIDE L Language Model

The TAIDE Project is committed to developing a generative AI dialogue engine model that suits the characteristics of Taiwan's language and culture, while constructing a trustworthy AI environment. By integrating the energy of industry, academia, and research, it promotes the development of trustworthy generative AI, enhances Taiwan's position in international competition, promotes industrial development, and avoids over - reliance on foreign technologies.

• The large - language models developed in this project are based on LLaMA2 - 7b released by Meta. By introducing available texts and training materials from different fields in Taiwan, the models' ability to respond in Traditional Chinese and performance in specific tasks are improved. The publicly released models are as follows:
  • [TAIDE - LX - 7B](https://huggingface.co/taide/TAIDE - LX - 7B): A model based on LLaMA2 - 7b, pre - trained (continuous pretraining) only with Traditional Chinese data. It is suitable for scenarios where users will further fine - tune the model. Since the pre - trained model has not been fine - tuned and preference - aligned, it may produce malicious or unsafe outputs. Please use it with caution.
  • [TAIDE - LX - 7B - Chat](https://huggingface.co/taide/TAIDE - LX - 7B - Chat): Based on TAIDE - LX - 7B, it enhances the ability to handle common office tasks and multi - round Q&A dialogues through instruction tuning. It is suitable for chat dialogues or task assistance. TAIDE - LX - 7B - Chat also provides a [4 - bit quantized model](https://huggingface.co/taide/TAIDE - LX - 7B - Chat - 4bit). The quantized model is mainly for user convenience, but it may affect performance and cause more unexpected problems. Please understand and note this.

🚀 Quick Start

You need to agree to the license terms before using this model.

License Information

✨ Features

• Enhanced Chinese Character Support: An additional 24,720 Chinese characters and words are added to strengthen the model's ability to process Traditional Chinese.
• High - quality Training Data: The training data of the model is strictly screened to improve the trustworthiness and applicability of the generated data.
• Optimized for Office Tasks: The model is enhanced for common office tasks such as automatic summarization, letter writing, article writing, Chinese - to - English translation, and English - to - Chinese translation.
• Cultural and Regional Knowledge: It has strengthened knowledge of Taiwan's local culture, language usage, and national conditions.
• Multi - round Q&A Capability: The model can handle multi - round question - answer dialogues.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

The example programs and documentation will be released on GitHub later.

Tokenizer Setup

tokenizer = AutoTokenizer.from_pretrained("taide/TAIDE-LX-7B-Chat", use_fast=False)

Prompt Templates

General Q&A

f"<s>[INST] {question} [/INST]"
# Replace {question} with the user's input.

With System Prompt

f"<s>[INST] <<SYS>>\n{sys}\n<</SYS>>\n\n{question} [/INST]"
# Replace {sys} with instructions, e.g., "You are an AI assistant from Taiwan named TAIDE, willing to help users from a Taiwanese perspective and answer questions in Traditional Chinese."
# Replace {question} with the user's question.

Multi - round Q&A

f"<s>[INST] <<SYS>>\n{sys}\n<</SYS>>\n\n{question1} [/INST] {model_answer_1} </s><s>[INST] {question2} [/INST]"
# Replace {sys} with instructions.
# Replace {question1} with the user's first question.
# Replace {model_answer_1} with the model's first answer.
# Replace {question2} with the user's second question.

Huggingface Chat Templates

General Q&A

chat = [
    {"role": "user", "content": "{question}"},
]
prompt = tokenizer.apply_chat_template(chat)
# Replace {question} with the user's input.

With System Prompt

chat = [
    {"role": "system", "content": "{sys}"},
    {"role": "user", "content": "{question}"},
]
prompt = tokenizer.apply_chat_template(chat)
# Replace {sys} with instructions, e.g., "You are an AI assistant from Taiwan named TAIDE, willing to help users from a Taiwanese perspective and answer questions in Traditional Chinese."
# Replace {question} with the user's question.

Multi - round Q&A

chat = [
    {"role": "system", "content": "{sys}"},
    {"role": "user", "content": "{question1}"},
    {"role": "assistant", "content": "{model_answer_1}"},
    {"role": "user", "content": "{question2}"},
]
prompt = tokenizer.apply_chat_template(chat)
# Replace {sys} with instructions.
# Replace {question1} with the user's first question.
# Replace {model_answer_1} with the model's first answer.
# Replace {question2} with the user's second question.

📚 Documentation

Model Parameters

Application Examples

🔧 Technical Details

Training Methods

Hardware and Software Specifications

• National Center for High - performance Computing H100
• Training Framework: PyTorch

Data Pre - processing

• Character Standardization: Standardize characters.
• Duplicate Removal: Remove duplicate data.
• Noise Removal:
  • Remove HTML tags and JavaScript from web data.
  • Remove non - standard characters or garbled characters.
  • Remove articles with too few words.
  • Remove specific formats in articles, such as line breaks added for typesetting.
• Personal Information Removal: Remove personal information such as emails and phone numbers.
• Inappropriate Text Removal: Remove inappropriate text such as gambling and pornographic content.

Character and Word Expansion

• To enhance the performance of Traditional Chinese input and output, the expanded data includes the following two parts:
  • Obtain Chinese characters from the [Standard Chinese Characters List of the Ministry of Education Variant Characters Dictionary](https://dict.variants.moe.edu.tw/appendix.jsp?ID = 1&ID = 0).
  • Extract 5 million sentences (2.1G) with more than 100 characters from Traditional Chinese Wikipedia, news, and Chinese common crawl data to train the tokenizer for Chinese words.

Continuous Pretraining (CP)

• Supplement a large amount of reliable Traditional Chinese knowledge.
• Hyperparameters:
  • Optimizer: AdamW
  • Learning Rate: 1e - 4
  • Batch Size: 1M tokens
  • Epoch: 1

Fine - Tuning (FT)

• Enable the model to answer questions in response to Traditional Chinese queries.
• Hyperparameters:
  • Optimizer: AdamW
  • Learning Rate: 5e - 5
  • Batch Size: 256K tokens
  • Epoch: 3

Training Data

Continuous Pretraining Data (about 140G)

Fine - Tuning Data

The TAIDE team trained the Llama2 series of models to generate fine - tuning data. The generated tasks include single - round or multi - round Q&A dialogues on world knowledge, creative writing, general knowledge, translation, summarization, programming, and Taiwanese values, totaling 128K records. The fine - tuning data will be released later.

📄 License

• TAIDE L Models Community License Agreement

⚠️ Disclaimer

Due to the limitations of the design architecture of LLM models and the possible biases in the data, any responses from the language model do not represent the stance of TAIDE. A security protection mechanism needs to be added before use, and the response content may also contain incorrect information. Please do not fully trust it.

👥 Development Team

• https://taide.tw/index/teamList

🔗 Related Links

• TAIDE Official Website
• TAIDE Huggingface
• [TAIDE Github](https://github.com/taide - taiwan)
• Kuwa AI

📑 Citation

• TAIDE Official Website