Unieval Sum

🚀 Pre-trained Evaluator for Text Generation

This project presents a pre-trained evaluator introduced in the EMNLP 2022 paper, aiming to provide a more comprehensive and fine-grained evaluation for text generation systems.

🚀 Quick Start

This pre - trained evaluator is based on the research in the EMNLP 2022 paper. It addresses the limitations of existing automatic evaluation metrics in Natural Language Generation (NLG) by offering a multi - dimensional evaluation approach.

✨ Features

Multi - dimensional Evaluation

Multi - dimensional evaluation is the key concept in human evaluation for Natural Language Generation (NLG). It assesses the generated text from multiple explainable dimensions like coherence and fluency. However, current automatic evaluation in NLG mainly relies on similarity - based metrics (e.g., ROUGE, BLEU), which are insufficient to distinguish advanced generation models. To solve this problem, we propose UniEval.

Pre - trained Evaluator - unieval - sum

unieval - sum is a pre - trained evaluator for the text summarization task. It can evaluate model output from four dimensions:

• coherence
• consistency
• fluency
• relevance

Moreover, it can be adapted to new dimensions and generation tasks, such as evaluating naturalness and informativeness for data - to - text generation.

💻 Usage Examples

Basic Usage

For detailed usage instructions, please refer to our GitHub repository.

📚 Documentation

The paper Towards a Unified Multi - Dimensional Evaluator for Text Generation provides in - depth details about the theory and implementation of this pre - trained evaluator. You can read it for a more comprehensive understanding.