Distilbert PoliticalBias

🚀 DistilBERT-PoliticalBias

DistilBERT-PoliticalBias is a DistilBERT-based model fine-tuned to detect and reduce political bias in text. It combines diffusion techniques with knowledge distillation from a fine-tuned RoBERTa teacher model to achieve unbiased text representations.

🚀 Quick Start

To use this model, you can load it with the Transformers library:

from transformers import DistilBertForSequenceClassification, RobertaTokenizer

model = DistilBertForSequenceClassification.from_pretrained('cajcodes/DistilBERT-PoliticalBias')
tokenizer = RobertaTokenizer.from_pretrained('cajcodes/DistilBERT-PoliticalBias')

✨ Features

• Novel Approach: Treats bias as "noise" and uses diffusion process to eliminate it.
• Knowledge Distillation: Aligns student model's predictions with the less biased outputs of the teacher model.

📦 Installation

No specific installation steps are provided in the original README other than loading the model and tokenizer using the Transformers library as shown in the "Usage" section.

💻 Usage Examples

Basic Usage

from transformers import DistilBertForSequenceClassification, RobertaTokenizer

model = DistilBertForSequenceClassification.from_pretrained('cajcodes/DistilBERT-PoliticalBias')
tokenizer = RobertaTokenizer.from_pretrained('cajcodes/DistilBERT-PoliticalBias')

Advanced Usage

import torch

sample_text = "We need to significantly increase social spending because it will reduce poverty and improve quality of life for all."
inputs = tokenizer(sample_text, return_tensors='pt')
outputs = model(**inputs)
predictions = torch.softmax(outputs.logits, dim=-1)
print(predictions)

📚 Documentation

Training

The model was trained using a synthetic dataset of 658 statements, each rated for bias. These statements were generated by GPT-4, covering a spectrum from highly conservative to highly liberal. The training process involved 21 epochs with a learning rate of 6e-6. The model was optimized using a combination of cross-entropy and KL divergence losses, with temperature scaling to distill knowledge from the teacher model.

Novel Approach

The training leverages a novel approach where bias is treated as "noise" that the diffusion process aims to eliminate. By using knowledge distillation, the student model learns to align its predictions with the less biased outputs of the teacher model, effectively reducing bias in the resulting text.

Evaluation

The model achieved the following performance metrics on the validation set:

• Matthews Correlation Coefficient (MCC): 0.593
• ROC AUC Score: 0.924

These metrics indicate a strong ability to classify and reduce bias in text.

🔧 Technical Details

The model employs a novel approach combining diffusion techniques with knowledge distillation from a fine-tuned RoBERTa teacher model. The training uses a synthetic dataset of 658 statements generated by GPT-4. The optimization process involves 21 epochs with a learning rate of 6e-6 and a combination of cross-entropy and KL divergence losses with temperature scaling.

📄 License

This project is licensed under the MIT license.

📦 Dataset

The dataset used for training, cajcodes/political-bias, contains 658 statements with bias ratings generated by GPT-4. The dataset is available for further analysis and model training.

📖 Citation

If you use this model or dataset, please cite as follows:

@misc{cajcodes_distilbert_political_bias,
  author = Christopher Jones,
  title = {DistilBERT-PoliticalBias: A Novel Approach to Detecting and Reducing Political Bias in Text},
  year = {2024},
  howpublished = {\url{https://huggingface.co/cajcodes/DistilBERT-PoliticalBias}},
}

📊 Information Table