Flan T5 Tsa Thor Xl

🚀 Model Card for Model ID

This model card provides detailed information about a fine - tuned Flan - T5 model for Target Sentiment Analysis (TSA) task. It offers insights into model details, usage scenarios, training procedures, and evaluation results.

✨ Features

• Chain - of - Thought Tuned: Based on the Chain - of - Thought approach for more accurate sentiment analysis.
• Multilingual Adaptation: Trained on translated data, suitable for English text analysis.
• Three - Hop - Reasoning Framework: Utilizes a novel framework for training and inference.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoTokenizer, T5ForConditionalGeneration

# Setup model path.
model_path = "nicolay-r/flan-t5-tsa-thor-xl"
# Setup device.
device = "cuda:0"

model = T5ForConditionalGeneration.from_pretrained(model_path, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
model.to(device)

def ask(prompt):
  inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
  inputs.to(device)
  output = model.generate(**inputs, temperature=1)
  return tokenizer.batch_decode(output, skip_special_tokens=True)[0]

def target_sentiment_extraction(sentence, target):
  # Setup labels.
  labels_list = ['neutral', 'positive', 'negative']
  # Setup Chain-of-Thought
  step1 = f"Given the sentence {sentence}, which specific aspect of {target} is possibly mentioned?"
  aspect = ask(step1)
  step2 = f"{step1}. The mentioned aspect is about {aspect}. Based on the common sense, what is the implicit opinion towards the mentioned aspect of {target}, and why?"
  opinion = ask(step2)
  step3 = f"{step2}. The opinion towards the mentioned aspect of {target} is {opinion}. Based on such opinion, what is the sentiment polarity towards {target}?"
  emotion_state = ask(step3)
  step4 = f"{step3}. The sentiment polarity is {emotion_state}. Based on these contexts, summarize and return the sentiment polarity only, " + "such as: {}.".format(", ".join(labels_list))
  # Return the final response.
  return ask(step4)

# Input sentence.
sentence = "Over the past 28 years, the leader has been working hard to achieve the release of Peltier and is a member of the Leonard Peltier Defense Committee."
# Input target.
target = "Peltier"
# output response
flant5_response = target_sentiment_extraction(sentence, target)
print(f"Author opinion towards `{target}` in `{sentence}` is:\n{flant5_response}")

Advanced Usage

python thor_finetune.py -m "nicolay-r/flan-t5-tsa-thor-xl" -r "thor" -d "rusentne2023" -z -bs 4 -f "./config/config.yaml"

📚 Documentation

Model Details

Update February 23 2025: 🔥 BATCHING MODE SUPPORT. See 🌌 Flan - T5 provider for bulk - chain project. Test is available here

This model represents a Chain - of - Thought tuned version Flan - T5 on Target Sentiment Analysis (TSA) task, using training data of RuSentNE - 2023 collection.

This model is designed for texts written in English. Since the original collection represents non - English texts, the content has been automatically translated into English using [googletrans].

For the given input sentence and mentioned entity in it (target), this model predicts the author's state by answering one of the following classes: [positive, negative, neutral]

Model Description

Model Sources

Uses

Direct Use

This sequence of scripts represents a purely torch and transformers based model usage for inference.

This example is also available on GoogleColab

Downstream Use

Please refer to the related section of the Reasoning - for - Sentiment - Analysis Framework

With this example, it applies this model in the THoR mode to the validation data of the RuSentNE - 2023 competition for evaluation.

Out - of - Scope Use

This model represents a fine - tuned version of the Flan - T5 on the RuSentNE - 2023 dataset. Since the dataset represents three - scale output answers (positive, negative, neutral), the behavior in general might be biased towards this particular task.

Recommendations

⚠️ Important Note

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information is needed for further recommendations.

Training Details

Training Data

We utilize train data which was automatically translated into English using GoogleTransAPI. The initial source of the texts written in Russian is from the following repository: https://github.com/dialogue-evaluation/RuSentNE-evaluation

The translated version of the dataset in English could be automatically downloaded via the following script: https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/rusentne23_download.py

Training Procedure

This model has been trained using the Three - hop - Reasoning framework, proposed in the paper: https://arxiv.org/abs/2305.11255

For training procedure accomplishment, the reforged version of this framework was used: https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework

Google - colab notebook for reproduction: https://colab.research.google.com/github/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/Reasoning_for_Sentiment_Analysis_Framework.ipynb

Setup: Flan - T5 - xl up to 64 tokens, 4 - batch size.

GPU: NVidia - A100, bfloat16, ~ 30 min/epoch

The overall training process took 3 epochs.


Training Hyperparameters

• Training regime: All the configuration details were highlighted in the related config file

Evaluation

Testing Data, Factors & Metrics

Testing Data

The direct link to the test evaluation data: https://github.com/dialogue-evaluation/RuSentNE-evaluation/blob/main/final_data.csv

Metrics

For the model evaluation, two metrics were used:

1. F1_PN -- F1 - measure over positive and negative classes;
2. F1_PN0 -- F1 - measure over positive, negative, and neutral classes;

Results

The test evaluation for this model showcases the F1_PN = 60.024

Below is the log of the training process that showcases the final performance on the RuSentNE - 2023 test set after 4 epochs (lines 5 - 6):

  F1_PN  F1_PN0  default   mode
0  66.678  73.838   73.838  valid
1  68.019  74.816   74.816  valid
2  67.870  74.688   74.688  valid
3  65.090  72.449   72.449   test
4  65.090  72.449   72.449   test

📄 License

The model is licensed under [Apache License 2.0](https://github.com/scofield7419/THOR - ISA/blob/main/LICENSE.txt)