Chinese ELECTRA Open-Source Pre-trained Model - Small Number of Parameters, High Performance, Empowering Chinese Text Processing

Chinese Electra Base Generator

Developed by hfl

Chinese ELECTRA is a pre-trained model developed by the Harbin Institute of Technology-iFLYTEK Joint Lab (HFL) based on the ELECTRA model released by Google and Stanford University. It features a small parameter size and high performance.

Large Language Model

Transformers

ChineseOpen Source License:Apache-2.0 #Lightweight and Efficient #Chinese Pre-training #Fill Mask

Downloads 15

Release Time : 3/2/2022

Model Overview

ELECTRA is an efficient pre-training model that significantly improves training efficiency by replacing BERT's masked language model task with a discriminative task. The Chinese ELECTRA series of models perform excellently in various NLP tasks, with a parameter size only 1/10 of BERT.

Model Features

Efficient Pre-training

Adopts the generator-discriminator architecture of ELECTRA, significantly improving training efficiency compared to traditional BERT models.

Small Parameter Size with High Performance

ELECTRA-small requires only 1/10 the parameters of BERT and its variants while achieving similar or even higher performance.

Chinese Optimization

Specifically optimized for Chinese language characteristics, excelling in Chinese NLP tasks.

Model Capabilities

Text Understanding

Text Generation

Semantic Analysis

Use Cases

Natural Language Processing

Text Classification

Can be used for sentiment analysis, topic classification, and other text classification tasks.

Question Answering System

Suitable for building Chinese question answering systems.

Named Entity Recognition

Can be used for Chinese named entity recognition tasks.

🚀 Chinese ELECTRA

This project releases Chinese ELECTRA models, which have a compact size and competitive performance compared to BERT and its variants, accelerating the research of Chinese pre - trained models.

🚀 Quick Start

Please use ElectraForPreTraining for discriminator and ElectraForMaskedLM for generator if you are re - training these models.

Google and Stanford University released a new pre - trained model called ELECTRA, which has a much compact model size and relatively competitive performance compared to BERT and its variants. For further accelerating the research of the Chinese pre - trained model, the Joint Laboratory of HIT and iFLYTEK Research (HFL) has released the Chinese ELECTRA models based on the official code of ELECTRA. ELECTRA - small could reach similar or even higher scores on several NLP tasks with only 1/10 parameters compared to BERT and its variants.

This project is based on the official code of ELECTRA: https://github.com/google-research/electra

📄 License

This project is under the Apache - 2.0 license.

📚 Documentation

Citation

If you find our resource or paper is useful, please consider including the following citation in your paper.

https://arxiv.org/abs/2004.13922

@inproceedings{cui-etal-2020-revisiting,
    title = "Revisiting Pre-Trained Models for {C}hinese Natural Language Processing",
    author = "Cui, Yiming  and
      Che, Wanxiang  and
      Liu, Ting  and
      Qin, Bing  and
      Wang, Shijin  and
      Hu, Guoping",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.findings-emnlp.58",
    pages = "657--668",
}

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご