Taiwan LLaMa V1.0 GGUF

🚀 Taiwan-LLaMa-v1.0 - GGUF

This repository contains GGUF format model files for Language Models for Taiwanese Culture v1.0, aiming to provide high - quality language processing capabilities for Taiwanese culture - related applications.

🚀 Quick Start

This repo contains GGUF format model files for Yen - Ting Lin's Language Models for Taiwanese Culture v1.0.

✨ Features

1. Traditional Mandarin Support

The model is fine - tuned to understand and generate text in Traditional Mandarin, making it suitable for Taiwanese culture and related applications.

2. Instruction - Tuned

Further fine - tuned on conversational data to offer context - aware and instruction - following responses.

3. Performance on Vicuna Benchmark

Taiwan - LLaMa's relative performance on Vicuna Benchmark is measured against models like GPT - 4 and ChatGPT. It's particularly optimized for Taiwanese culture.

4. Flexible Customization

Advanced options for controlling the model's behavior like system prompt, temperature, top - p, and top - k are available in the demo.

📦 Installation

How to deploy the model on my own machine?

We recommend hosting models with 🤗 Text Generation Inference. Please see their license for details on usage and limitations.

bash run_text_generation_inference.sh "yentinglin/Taiwan-LLaMa-v1.0" NUM_GPUS DIR_TO_SAVE_MODEL PORT MAX_INPUT_LEN MODEL_MAX_LEN

Setup development environment

conda create -n taiwan-llama python=3.10 -y 
conda activate taiwan-llama
pip install -r requirements.txt

💻 Usage Examples

Basic Usage

Prompt format follows vicuna - v1.1 template:

A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {user} ASSISTANT:

📚 Documentation

About GGUF

GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.

The key benefit of GGUF is that it is an extensible, future - proof format which stores more information about the model as metadata. It also includes significantly improved tokenization code, including for the first time full support for special tokens. This should improve performance, especially with models that use new special tokens and implement custom prompt templates.

As of August 25th, here is a list of clients and libraries that are known to support GGUF:

• llama.cpp.
• text - generation - webui, the most widely used web UI. Supports GGUF with GPU acceleration via the ctransformers backend - llama - cpp - python backend should work soon too.
• KoboldCpp, now supports GGUF as of release 1.41! A powerful GGML web UI, with full GPU accel. Especially good for story telling.
• LM Studio, version 0.2.2 and later support GGUF. A fully featured local GUI with GPU acceleration on both Windows (NVidia and AMD), and macOS.
• LoLLMS Web UI, should now work, choose the c_transformers backend. A great web UI with many interesting features. Supports CUDA GPU acceleration.
• ctransformers, now supports GGUF as of version 0.2.24! A Python library with GPU accel, LangChain support, and OpenAI - compatible AI server.
• llama - cpp - python, supports GGUF as of version 0.1.79. A Python library with GPU accel, LangChain support, and OpenAI - compatible API server.
• candle, added GGUF support on August 22nd. Candle is a Rust ML framework with a focus on performance, including GPU support, and ease of use.

Repositories available

• 2, 3, 4, 5, 6 and 8 - bit GGUF models for CPU+GPU inference
• 2, 3, 4, 5, 6 and 8 - bit GGML models for CPU+GPU inference (deprecated)
• Yen - Ting Lin's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions

Original model card: Yen - Ting Lin's Language Models for Taiwanese Culture v1.0

Overview

Taiwan - LLaMa is a full parameter fine - tuned model based on LLaMa 2 for Traditional Mandarin applications.

Taiwan - LLaMa v1.0 pretrained on over 5 billion tokens and instruction - tuned on over 490k conversations both in traditional mandarin.

Demo

A live demonstration of the model can be accessed at Hugging Face Spaces.

Work in progress

• [ ] Improved pretraining: A refined pretraining process (e.g. more data from Taiwan, training strategies) is under development, aiming to enhance model performance for better Taiwanese culture.
• [ ] Extend max length: Utilizing the Rope mechanism as described in the paper, the model's length will be extended from 4k to 8k.

Taiwanese Culture Examples

Model

We provide a number of model checkpoints that we trained. Please find them on Hugging Face here. Here are some quick links to the checkpoints that are finetuned from LLaMa 2:

Data

Here are some quick links to the datasets that we used to train the models:

Architecture

Taiwan - LLaMa is based on LLaMa 2, leveraging transformer architecture, flash attention 2, and bfloat16.

It includes:

• Pretraining Phase: Pretrained on a vast corpus of over 5 billion tokens, extracted from common crawl in Traditional Mandarin.
• Fine - tuning Phase: Further instruction - tuned on over 490k multi - turn conversational data to enable more instruction - following and context - aware responses.

Generic Capabilities on Vicuna Benchmark

The data is translated into traditional mandarin for evaluating the general capability.

The scores are calculated with ChatGPT as the baseline, represented as 100%. The other values show the relative performance of different models compared to ChatGPT.

🔧 Technical Details

Model Information

📄 License

The code in this project is licensed under the Apache 2.0 License - see the LICENSE file for details.

The models included in this project are licensed under the LLAMA 2 Community License.

Citations

If you use our code, data, or models in your research, please cite this repository. You can use the following BibTeX entry:

@inproceedings{lin-chen-2023-llm,
    title = "{LLM}-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models",
    author = "Lin, Yen-Ting  and Chen, Yun-Nung",
    booktitle = "Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvAI 2023)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.nlp4convai-1.5",
    pages = "47--58"
}

@misc{taiwanllama,
    author={Lin, Yen-Ting and Chen, Yun-Nung},
    title={Language Models for Taiwanese Culture},
    year={2023},
    url={https://github.com/MiuLab/Taiwan-LLaMa},
    note={Code and models available at https://github.com/MiuLab/Taiwan-LLaMa},
}

Collaborate With Us

If you are interested in contributing to the development of Traditional Mandarin language models, exploring new applications, or leveraging Taiwan - LLaMa for your specific needs, please don't hesitate to contact us. We welcome collaborations from academia, industry, and individual contributors.