language:

• vi
• en license: llama2 pipeline_tag: text-generation model-index:
• name: ToRoLaMa-7b-v1.0 results:
  • task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics:
    • type: acc_norm value: 51.71 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=allbyai/ToRoLaMa-7b-v1.0 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics:
    • type: acc_norm value: 73.82 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=allbyai/ToRoLaMa-7b-v1.0 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics:
    • type: acc value: 45.34 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=allbyai/ToRoLaMa-7b-v1.0 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics:
    • type: mc2 value: 44.89 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=allbyai/ToRoLaMa-7b-v1.0 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics:
    • type: acc value: 70.09 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=allbyai/ToRoLaMa-7b-v1.0 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics:
    • type: acc value: 1.36 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=allbyai/ToRoLaMa-7b-v1.0 name: Open LLM Leaderboard

ToRoLaMa: The Vietnamese Instruction-Following and Chat Model

Authors: Duy Quang Do¹, Hoang Le¹ and Duc Thang Nguyen^2
1Taureau AI, Hanoi, Vietnam
²Torus AI, Toulouse, France

ToRoLaMa is the result of a collaborative effort of Vietnam-based Taureau AI and France-based Torus AI. It stands as an open-source, multi-turn, large language model (LLM), initially created with a focus on the Vietnamese language. It represents the first step towards a wider goal of supporting a variety of international languages.

• Introduction
• Model weights
• Technical overview
• Evaluations
• License
• Disclaimer
• Acknowledgement

Introduction

Torus AI (official name: Torus Actions SAS) was founded in Toulouse (France) in 2019 by a group of scientists under the leadership of Nguyen Tien Zung, distinguished professor of mathematics at the University of Toulouse. The name Torus Actions comes from the toric conservation principle discovered by Zung:

Everything conserved by a dynamical system is also conserved by its associated torus actions.

Taureau AI, set up in 2021 in Hanoi by Torus AI people, is focused on the development of a general purpose AI platform, AI product engineering and software development, to serve the other companies inside and outside the Torus AI ecosystem.

Our common objective is to create augmented intelligence solutions that serve millions of people and make the world a happier place.

Our large language model - ToRoLaMa, developed using a diverse and extensive dataset, aims to provide an enhanced understanding and representation of languages, aspiring to meet and possibly exceed the efficiency, performance, and applicability of existing commercial LLMs.

With ToRoLaMa, we hope to contribute to the rapid progress in language processing for Vietnamese speaking people and applications. We also plan to extend it (and other LLMs) to other languages.

This release includes the model weights, inference code, and evaluation results for the 7B (7 billion parameter) version.

Model weights

Our lastest weights for ToRoLaMa can be found here:

Technical overview

The ToRoLaMa's pre-trained model is based on Vietnamese-LLaMA2, a fine-tuned version of LLaMA 2 model provided by bkai-foundation-labs, enhanced with a large Vietnamese-language dataset. The model then was trained using 430K high-quality, multi-turn questions/answers. Data sources for the training include UIT-ViQUAD, Bactrian-X, Grade-school-math, etc and our in-house data that contain conversations on multiple topics.

Key advantages of ToRoLaMa include:

• Open-source availability under the LLaMA 2 License
• Enhanced speed with a smaller model size and an innovative Vietnamese Tokenizer, whose tokens are 25% shorter compared to ChatGPT and LLaMA for Vietnamese phrases.
• Superior performance over existing open-source models (see benchmark results below).
• Simplified deployment for a wide range of applications.

Evaluations

We used benchmark results of Vicuna and PhoGPT to evaluate ToRoLaMa and compared our results with others using the Fastchat MT-bench method.The table below shows that ToRoLaMa performs competitively against state-of-the-art models like ChatGPT.

The Fastchat benchmark method, used for evaluating language models, primarily focuses on the accuracy of information in responses. However, an important aspect not accounted for in this method is the accuracy in the choice of language (English vs. Vietnamese). Both URA-LLaMA-7B and URA-LLaMA-13B often respond in English to Vietnamese questions. Their performance may be rated much lower when specifically benchmarked for proficiency in Vietnamese.

The benchmark scores are shown in the following table:

*: The scores of URA models here do not take into account the fact that they often answer in English to questions posed in Vietnamese.

The details of benchmark in terms of subjects are shown in the following figure (we do not display URA-LLaMA because they generate half of the answers in English):

The above benchmark results show that ToRoLaMa excels in qualitative tasks compared to the other models, particularly with its ability to write and answer almost on par with GPT-3.5-turbo. However, it shows limitations in quantitative tasks like coding and mathematics due to the nature of its training data. This suggests opportunities for future improvements in STEM-related tasks.

For detailed benchmark information and to rerun the evaluation code, refer to Fastchat MT-bench method. We have included the answers from each model, the prompts, and the evaluation results HERE for reproduction. The generated results can also be accessed HERE for human evaluation.

Run the model

ToRoLaMa uses a prompt format similar to Vicuna, designed for multi-turn, high-speed, and token-efficient conversations. An example prompt is shown below for illustration.

Cuộc hội thoại giữa người dùng và một trí thông minh nhân tạo. Đưa ra câu trả lời chính xác, giúp ích cho người dùng.

USER: chào
ASSISTANT: Xin chào! Tôi có thể giúp gì cho bạn hôm nay?</s>
USER: để quản lý thời gian hiệu quả tôi nên làm thế nào??
ASSISTANT: Để quản lý thời gian hiệu quả, bạn có thể thực hiện các bước sau đây:

    Lập kế hoạch: Tạo ra lịch trình hoặc danh sách công việc cần làm trong ngày hoặc tuần.

    Ưu tiên công việc: Xác định công việc quan trọng và ưu tiên thực hiện trước.

    Sử dụng lịch: Sử dụng lịch để theo dõi các hoạt động và sự kiện trong ngày.

    Sử dụng một số công cụ hỗ trợ: Sử dụng các công cụ trực tuyến, ứng dụng điện thoại di động để giúp quản lý thời gian.

    Tránh phân tâm: Tránh những việc phức tạp và không cần thiết để giải quyết trong quá trình làm việc.

    Nghỉ ngơi: Để giảm căng thẳng và tăng năng suất, hãy dành thời gian nghỉ ngơi giữa các công việc.

    Đánh giá và cải tiến: Đánh giá các phương pháp quản lý thời gian của bạn để tìm cách cải tiến trong tương lai.

Những cách trên sẽ giúp bạn quản lý thời gian hiệu quả hơn và tăng cường năng suất trong công việc.</s>

The file inference_hf.py in our github repository contains an example code for running ToRoLaMa model from Huggingface hub. Execute it using the following command:

python inference_hf.py

Deployment

ToRoLaMa can be easily deployed using Fastchat.

Step 1: Install fastchat

pip3 install "fschat[model_worker,webui]"

Step 2: Run the RESTful API Server

Begin by running the controller:

python3 -m fastchat.serve.controller

Next, launch the model worker:

python3 -m fastchat.serve.model_worker --model-path path-to-ToRoLaMa --conv-template vicuna_v1.1

Then, initiate the RESTful API server:

python3 -m fastchat.serve.openai_api_server --host localhost --port 8000

Finaly, run the example streamlit code:

streamlit run demo.py

License

ToRoLaMa is licensed under the ToRoLaMa community License agreement.

ToRoLaMa is licensed under the LLaMA 2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.

Disclaimer

This model is derived from Meta's LLaMA-2 model, and therefore strictly complies with the LLaMA 2 Community License Agreement. We explicitly declare that we offer no assurances, guarantees, or warranties about the accuracy, reliability, usability, or completeness of the model's outputs or the data presented therein. We disclaim all liability for any immediate or subsequent losses, damages or adverse consequences arising from the use of our model. Please be aware that the model's generated content might include inaccuracies, profanity, hate speech, discriminatory remarks, and/or misleading narratives. Using this model or its derivatives for commercial purposes requires full compliance with all applicable local laws and regulations regarding the legality of the content produced by the model. We hold no accountability for any products or services that are developed using ToRoLaMa and its related files.

Acknowledgement

The bkai-foundation-labs, and fastchat and references therein have been used in this work.

In case you use ToRoLaMa, please cite our work in your publications :

@misc{allbyai2023ToRoLaMa,
    title={ToRoLaMa: The Vietnamese Instruction-Following and Chat Model},
    author={Duy Quang Do, Hoang Le and Duc Thang Nguyen},
    year={2023},
    note={https://github.com/allbyai/ToRoLaMa}
    howpublished={Software}
}

Open LLM Leaderboard Evaluation Results

Detailed results can be found here