Electra Base Generator

🚀 ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

ELECTRA is a novel approach for self-supervised language representation learning. It enables pre - training of transformer networks with relatively low computational requirements. ELECTRA models are trained to differentiate between "real" input tokens and "fake" ones generated by another neural network, similar to the discriminator in a GAN. At a small scale, ELECTRA can yield strong results even when trained on a single GPU. At a large scale, it achieves state - of - the - art performance on the SQuAD 2.0 dataset.

🚀 Quick Start

For a detailed description and experimental results, please refer to our paper ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators.

This repository contains code for pre - training ELECTRA, including small ELECTRA models on a single GPU. It also supports fine - tuning ELECTRA on downstream tasks such as classification tasks (e.g., GLUE), QA tasks (e.g., SQuAD), and sequence tagging tasks (e.g., text chunking).

💻 Usage Examples

Basic Usage

from transformers import pipeline

fill_mask = pipeline(
    "fill-mask",
    model="google/electra-base-generator",
    tokenizer="google/electra-base-generator"
)

print(
    fill_mask(f"HuggingFace is creating a {fill_mask.tokenizer.mask_token} that the community uses to solve NLP tasks.")
)

📄 License

This project is licensed under the Apache 2.0 license.