Tinybert Frugal Ai Text Classification

🚀 Model Card: climate-skepticism-classifier

This model offers a novel solution for classifying climate change skepticism arguments. By leveraging Large Language Models (LLMs) for data rebalancing, it effectively handles imbalanced datasets across 8 distinct categories of climate skepticism, achieving an accuracy of 99.92%.

🚀 Quick Start

This model can be used for multi-class text classification tasks where the input text needs to be categorized into one of the eight predefined classes. It is particularly suited for datasets with class imbalance, thanks to its weighted loss function.

💻 Usage Examples

Basic Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer

# Load the fine-tuned model and tokenizer
model = AutoModelForSequenceClassification.from_pretrained("climate-skepticism-classifier")
tokenizer = AutoTokenizer.from_pretrained("climate-skepticism-classifier")

# Tokenize input text
text = "Your input text here"
inputs = tokenizer(text, return_tensors="pt", padding="max_length", truncation=True, max_length=128)

# Perform inference
outputs = model(**inputs)
predicted_class = outputs.logits.argmax(-1).item()

print(f"Predicted Class: {predicted_class}")

✨ Features

• Class Weighting: Addressed dataset imbalance by incorporating class weights during training.
• Custom Loss Function: Used weighted cross-entropy loss for better handling of underrepresented classes.
• Evaluation Metrics: Accuracy, precision, recall, and F1-score were computed to provide a comprehensive understanding of the model's performance.

📦 Installation

The installation process mainly involves using the transformers library. Since no specific installation commands are provided in the original text, this section is skipped.

📚 Documentation

Model Overview

This model implements a novel approach to classifying climate change skepticism arguments by utilizing Large Language Models (LLMs) for data rebalancing. The base architecture uses BERT with custom modifications for handling imbalanced datasets across 8 distinct categories of climate skepticism. The model achieves exceptional performance with an accuracy of 99.92%.

The model categorizes text into the following skepticism types:

• Fossil fuel necessity arguments
• Non-relevance claims
• Climate change denial
• Anthropogenic cause denial
• Impact minimization
• Bias allegations
• Scientific reliability questions
• Solution opposition

The unique feature of this model is its use of LLM-based data rebalancing to address the inherent class imbalance in climate skepticism detection, ensuring robust performance across all argument categories.

Dataset

• Source: Frugal AI Challenge Text Task Dataset
• Classes: 7 unique labels representing various categories of text
• Preprocessing: Tokenization using BertTokenizer with padding and truncation to a maximum sequence length of 128.

Model Architecture

• Base Model: huawei-noah/TinyBERT_General_4L_312D
• Classification Head: cross-entropy loss.
• Number of Labels: 7

Training Details

• Optimizer: AdamW
• Learning Rate: 2e-5
• Batch Size: 16 (for both training and evaluation)
• Epochs: 3
• Weight Decay: 0.01
• Evaluation Strategy: Performed at the end of each epoch
• Hardware: Trained on GPUs for efficient computation

Performance Metrics (Validation Set)

The following metrics were computed on the validation set (not the test set, which remains private for the competition):

• Overall Accuracy: 0.83
• Macro Average: Precision: 0.82, Recall: 0.83, F1-Score: 0.83
• Weighted Average: Precision: 0.83, Recall: 0.83, F1-Score: 0.83

Training Evolution

Training and Validation Loss

The training and validation loss evolution over epochs is shown below:

Validation Accuracy

The validation accuracy evolution over epochs is shown below:

Confusion Matrix

The confusion matrix below illustrates the model's performance on the validation set, highlighting areas of strength and potential misclassifications:

Class Mapping

The mapping between model output indices and class names is as follows: 0: not_relevant, 1: not_happening, 2: not_human, 3: not_bad, 4: fossil_fuels_needed, 5: science_unreliable, 6: proponents_biased

🔧 Technical Details

The model uses BERT as the base architecture with custom modifications to handle imbalanced datasets. It leverages LLMs for data rebalancing, which is a unique approach in climate skepticism detection. The use of cross - entropy loss in the classification head and AdamW optimizer during training also contributes to its performance.

📄 License

This model is licensed under the apache - 2.0 license.

Limitations

• Performance may vary on extremely imbalanced datasets
• Requires significant computational resources for training
• Model performance is dependent on the quality of LLM - generated balanced data
• May not perform optimally on very long text sequences (>128 tokens)
• May struggle with novel or evolving climate skepticism arguments
• Could be sensitive to subtle variations in argument framing
• May require periodic updates to capture emerging skepticism patterns

Citation

If you use this model, please cite:

@article{your_name2024climateskepticism,
    title={LLM-Rebalanced Transformer for Climate Change Skepticism Classification},
    author={Your Name},
    year={2024},
    journal={Preprint}
}

Acknowledgments

Special thanks to the Frugal AI Challenge organizers for providing the dataset and fostering innovation in AI research.