RoLlama3-8b-Instruct-2024-06-28 Open Source Model - A Text Generation Tool Optimized for Romanian Language Tasks

Rollama3 8b Instruct 2024 06 28

Developed by OpenLLM-Ro

RoLlama3 is a series of Romanian language generation text models based on Meta Llama 3, this is the 8B parameter instruction fine-tuned version, specifically optimized for Romanian language tasks.

Large Language Model

Transformers

Other#Romanian language generation #Multi-task instruction #Academic evaluation optimization

Downloads 62

Release Time : 6/6/2024

Model Overview

This model is a Romanian large language model developed by the OpenLLM-Ro project, fine-tuned on Meta-Llama-3-8B, suitable for various Romanian natural language processing tasks, especially assistant-like chat scenarios.

Model Features

Romanian language optimization

Specifically pre-trained and fine-tuned for Romanian language, excelling in Romanian language tasks

Multi-dataset fine-tuning

Instruction fine-tuned using 7 different Romanian language datasets, covering a wide range of application scenarios

Comprehensive evaluation

Thoroughly evaluated on multiple Romanian benchmarks, including RoMT-Bench, RoCulturaBench, etc.

Model Capabilities

Romanian text generation

Question answering systems

Machine translation

Text classification

Semantic similarity calculation

Use Cases

Education

Academic Q&A

Answering Romanian academic questions

50.56% accuracy on Romanian_Academic_Benchmarks

Translation

English-Romanian translation

Machine translation between English and Romanian

WMT_EN-RO BLEU 24.01, WMT_RO-EN BLEU 27.36

Sentiment analysis

Romanian sentiment classification

Sentiment analysis on Romanian text

LaRoSeDa_binary macro F1 97.52%

🚀 OpenLLM-Ro/RoLlama3-8b-Instruct-2024-06-28

This project is based on the meta-llama/Meta-Llama-3-8B model, trained on multiple Romanian datasets. It provides performance metrics on various text - generation tasks.

📚 Documentation

Model Information

Property	Details
Base Model	meta-llama/Meta-Llama-3-8B
Training Datasets	OpenLLM-Ro/ro_sft_alpaca, OpenLLM-Ro/ro_sft_alpaca_gpt4, OpenLLM-Ro/ro_sft_dolly, OpenLLM-Ro/ro_sft_selfinstruct_gpt4, OpenLLM-Ro/ro_sft_norobots, OpenLLM-Ro/ro_sft_orca, OpenLLM-Ro/ro_sft_camel

Performance Metrics

The model has been evaluated on multiple text - generation tasks. Here are some of the key results:

RoMT - Bench

Metric	Value
Score	5.15
First turn	6.03
Second turn	4.28

RoCulturaBench

Metric	Value
Score	3.71

Romanian_Academic_Benchmarks

Metric	Value
Average accuracy	50.56

OpenLLM - Ro/ro_arc_challenge

Metric	0 - shot	1 - shot	3 - shot	5 - shot	10 - shot	25 - shot
Accuracy	41.90	44.30	44.56	45.50	46.10	45.84
Average accuracy	44.70

OpenLLM - Ro/ro_mmlu

Metric	0 - shot	1 - shot	3 - shot	5 - shot
Accuracy	50.85	51.24	53.30	53.39
Average accuracy	52.19

OpenLLM - Ro/ro_winogrande

Metric	0 - shot	1 - shot	3 - shot	5 - shot
Accuracy	65.19	66.54	67.88	69.30
Average accuracy	67.23

OpenLLM - Ro/ro_hellaswag

Metric	0 - shot	1 - shot	3 - shot	5 - shot	10 - shot
Accuracy	56.12	57.37	57.92	58.18	58.85
Average accuracy	57.69

OpenLLM - Ro/ro_gsm8k

Metric	1 - shot	3 - shot	5 - shot
Accuracy	29.42	30.02	31.24
Average accuracy	30.23

LaRoSeDa_binary

Metric	0 - shot	1 - shot	3 - shot	5 - shot
Macro - f1	97.43	96.60	97.90	98.13
Average macro - f1	97.52

LaRoSeDa_multiclass

Metric	0 - shot	1 - shot	3 - shot	5 - shot
Macro - f1	63.77	68.91	66.36	70.61
Average macro - f1	67.41

LaRoSeDa_binary_finetuned

Metric	Value
Average macro - f1	94.15

LaRoSeDa_multiclass_finetuned

Metric	Value
Average macro - f1	87.13

WMT_EN - RO

Metric	0 - shot	1 - shot	3 - shot	5 - shot
Bleu	6.92	29.33	29.79	30.02
Average bleu	24.01

WMT_RO - EN

Metric	0 - shot	1 - shot	3 - shot	5 - shot
Bleu	4.50	30.30	36.96	37.70
Average bleu	27.36

WMT_EN - RO_finetuned

Metric	Value
Average bleu	26.53

WMT_RO - EN_finetuned

Metric	Value
Average bleu	40.36

XQuAD

Metric	0 - shot	1 - shot	3 - shot	5 - shot
Exact_match	4.45	48.24	52.03	53.03
F1	26.08	68.40	71.92	71.60
Average exact_match	39.43
Average f1	59.50

XQuAD_finetuned

Metric	Value
Average exact_match	44.45
Average f1	59.76

STS

Metric	1 - shot	3 - shot	5 - shot
Spearman	77.76	76.72	77.12
Pearson	77.87	85.80	86.05
Average spearman	77.20
Average pearson	77.87

📄 License

The model is licensed under the CC - BY - NC - 4.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご