Poro 34B Chat

🚀 Poro 34B Chat

Poro 34B Chat is a chat-tuned version of Poro 34B. It is trained to follow instructions in both Finnish and English. Quantized versions are available on Poro 34B-chat-GGUF.

Due to the limited amount of instruction tuning available for Finnish, documents from the English datasets were machine-translated by the Poro 34B base model into Finnish and then used to train this chat version. We only selected datasets that are available for commercial use and only contain synthetic data gathered in a ToS-compliant manner.

More information about the data selection and translation process for our Finnish dataset can be found on the LumiOpen/instruction-collection-fin page.

Poro was created through a collaboration between SiloGen from Silo AI, the TurkuNLP group of the University of Turku, and High Performance Language Technologies (HPLT). Training was conducted on the LUMI supercomputer, using compute resources generously provided by CSC - IT Center for Science, Finland.

This project is part of an ongoing effort to create open-source large language models for non-English and especially low-resource languages like Finnish. By combining English and Finnish training data, we get a model that outperforms previous Finnish-only models. It is also fluent in English and code and capable of basic translation between English and Finnish.

🚀 Quick Start

The Poro 34B Chat model is ready to use for Finnish and English instruction-following tasks. You can access the quantized versions on Poro 34B-chat-GGUF.

✨ Features

• Multilingual Support: Capable of following instructions in both Finnish and English, and performing basic translation between the two languages.
• Code Fluency: Fluent in English and code, suitable for various programming-related tasks.
• Performance: Outperforms previous Finnish-only models through the combination of English and Finnish training data.

🔧 Technical Details

Fine Tuning

Poro-34b-Chat is an SFT finetune of Poro-34b on a collection of Finnish and English instruction datasets. The collection consists of roughly 40% English, 40% Finnish, and 20% cross-lingual entries.

We finetuned the base model for 3 epochs with a learning rate of 2e-05, a warmup ratio of 0.1, and a global batch size of 48. For full-parameter finetuning, we used 3 nodes (8 GPUs per node). We used the Alignment Handbook code for finetuning.

📦 Datasets

Finnish and Cross-lingual

• LumiOpen/instruction-collection-fin

English

• databricks-dolly-15k
• Curated OASST2
• Argilla/10k_prompts_ranked_mistral_large_responses

💻 Usage Examples

Chat template

We use the ChatML chat template. For example:

<|im_start|>system 
You can add an optional system prompt here.<|im_end|> 
<|im_start|>user 
Miten rakennan tietokoneen?<|im_end|>
<|im_start|>assistant 

📚 Documentation

Evaluations

We relied on the popular MTBench benchmark to evaluate multi-turn performance.

Since MTBench is an English-only benchmark, we also release this fork of MTBench Finnish with multilingual support and machine-translated Finnish prompts. Our scores for both benchmarks are as follows:

Note: Updated on 18 June 2024

📄 License

Poro 34B chat is released under the Apache 2.0 license.

📚 Citation

@misc{luukkonen2024poro,
      title={Poro 34B and the Blessing of Multilinguality},
      author={Risto Luukkonen and Jonathan Burdge and Elaine Zosa and Aarne
Talman and Ville Komulainen and Väinö Hatanpää and Peter Sarlin and Sampo
Pyysalo},
      year={2024},
      eprint={2404.01856},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}