Electra Large Generator

🚀 ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

ELECTRA is a novel approach for self-supervised language representation learning. It enables pre - training of transformer networks with relatively low computational resources. ELECTRA models are trained to differentiate between "real" input tokens and "fake" ones generated by another neural network, similar to the discriminator in a GAN. At a small scale, ELECTRA can yield strong results even when trained on a single GPU. At a large scale, it achieves state - of - the - art performance on the SQuAD 2.0 dataset.

🚀 Quick Start

This repository offers code for pre - training ELECTRA, including small ELECTRA models that can be trained on a single GPU. It also supports fine - tuning ELECTRA on various downstream tasks such as classification tasks (e.g., GLUE), QA tasks (e.g., SQuAD), and sequence tagging tasks (e.g., text chunking).

For a detailed description and experimental results, please refer to our paper ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators.

✨ Features

• Low - compute Pre - training: ELECTRA can pre - train transformer networks with relatively little compute.
• Scalable Performance: It achieves strong results at a small scale (single - GPU training) and state - of - the - art results at a large scale (e.g., on SQuAD 2.0).
• Downstream Task Support: Supports fine - tuning on a variety of downstream tasks including classification, QA, and sequence tagging.

💻 Usage Examples

Basic Usage

from transformers import pipeline

fill_mask = pipeline(
    "fill-mask",
    model="google/electra-large-generator",
    tokenizer="google/electra-large-generator"
)

print(
    fill_mask(f"HuggingFace is creating a {nlp.tokenizer.mask_token} that the community uses to solve NLP tasks.")
)

📄 License

This project is licensed under the Apache 2.0 license.