Alignscorecs

🚀 AlignScoreCS

A MultiTask multilingual model designed to assess factual consistency in context-claim pairs across diverse Natural Language Understanding (NLU) tasks.

This multi-task multilingual model, AlignScoreCS, is crafted to evaluate factual consistency in context-claim pairs across a wide array of Natural Language Understanding (NLU) tasks. These tasks include Summarization, Question Answering (QA), Semantic Textual Similarity (STS), Paraphrase, Fact Verification (FV), and Natural Language Inference (NLI).

AlignScoreCS is fine-tuned on a massive multi-task dataset that consists of 7 million documents, covering these NLU tasks in both Czech and English. Thanks to its multilingual pre-training, it holds the potential to be used in numerous other languages. The architecture can handle tasks through regression, binary classification, or ternary classification. However, for evaluation, we suggest using the AlignScore function.

This work draws inspiration from its English counterpart AlignScore: Evaluating Factual Consistency with a Unified Alignment Function. Nevertheless, during training, we used homogeneous batches instead of heterogeneous ones and employed three distinct architectures sharing a single encoder. This setup allows each architecture to be used independently with its classification head.

🚀 Quick Start

This section provides an overview of how to quickly get started with the AlignScoreCS model.

✨ Features

• Multilingual Capability: Can assess factual consistency in Czech and English, with potential for other languages.
• Multi - Task Adaptability: Suitable for various NLU tasks such as Summarization, QA, STS, etc.
• Flexible Classification: Supports regression, binary classification, and ternary classification.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

  # Assuming you copied the attached Files_and_versions/AlignScore.py file for ease of use in transformers.
  from AlignScoreCS import AlignScoreCS
  alignScoreCS = AlignScoreCS.from_pretrained("krotima1/AlignScoreCS")
  # put the model to cuda to accelerate
  print(alignScoreCS.score(context="This is context", claim="This is claim"))

📚 Documentation

Evaluation

Similar to the paper AlignScore, we use their AlignScore function. This function chunks the context into approximately 350 tokens and splits the claim into sentences. Each context chunk is evaluated against each claim sentence, and a single consistency score is aggregated.

The AlignScoreCS model is built on three XLM - RoBERTa architectures that share one encoder. It is a multi - task multilingual model for assessing facticity in various NLU tasks in Czech and English. We followed the initial paper AlignScore https://arxiv.org/abs/2305.16739. We trained a model using a shared architecture of checkpoint xlm - roberta - large [xlm - roberta](https://huggingface.co/FacebookAI/xlm - roberta - large) with three linear layers for regression, binary classification, and ternary classification.

Training datasets

The following table presents the datasets used for training the model. We translated these English datasets to Czech using seamLessM4t.

📄 License

This work is licensed under CC - BY - 4.0.