Chinese-macbert-base Open-source Language Model - Alleviate Pretraining Differences and Boost Text Processing Optimization

Chinese Macbert Base

Developed by hfl

MacBERT is an improved BERT model that uses a novel MLM as a corrective masked language model pre-training task, alleviating the discrepancy between pre-training and fine-tuning stages.

Large Language Model ChineseOpen Source License:Apache-2.0 #Corrective Masked Pre-training #Chinese NLP Optimization #Whole Word Masking Technique

Downloads 22.48k

Release Time : 3/2/2022

Model Overview

MacBERT is an enhanced Chinese BERT model that uses similar words for masking instead of the traditional [MASK] token, reducing the gap between pre-training and fine-tuning phases.

Model Features

Corrective MLM

Uses similar words for masking instead of the traditional [MASK] token, reducing the gap between pre-training and fine-tuning stages.

Whole Word Masking

Employs whole word masking technology to enhance the model's understanding of complete vocabulary.

N-gram Masking

Supports N-gram masking to strengthen the model's ability to process continuous vocabulary.

Sentence Order Prediction

Integrates Sentence Order Prediction (SOP) tasks to improve the model's understanding of sentence structure.

Model Capabilities

Text Understanding

Text Generation

Text Classification

Named Entity Recognition

Question Answering System

Use Cases

Natural Language Processing

Chinese Text Classification

Used for tasks such as sentiment analysis and topic classification in Chinese text.

Named Entity Recognition

Identifies entities such as person names, place names, and organization names in Chinese text.

Question Answering System

Builds a Chinese question answering system to respond to user queries.

🚀 MacBERT

MacBERT is an improved BERT with a novel MLM as correction pre - training task, which helps to mitigate the discrepancy between pre - training and fine - tuning.

🚀 Quick Start

Please use 'Bert' related functions to load this model!

This repository contains the resources in our paper "Revisiting Pre-trained Models for Chinese Natural Language Processing", which will be published in "Findings of EMNLP". You can read our camera-ready paper through ACL Anthology or arXiv pre-print.

Revisiting Pre-trained Models for Chinese Natural Language Processing
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, Guoping Hu

You may also interested in,

Chinese BERT series: https://github.com/ymcui/Chinese-BERT-wwm
Chinese ELECTRA: https://github.com/ymcui/Chinese-ELECTRA
Chinese XLNet: https://github.com/ymcui/Chinese-XLNet
Knowledge Distillation Toolkit - TextBrewer: https://github.com/airaria/TextBrewer

More resources by HFL: https://github.com/ymcui/HFL-Anthology

✨ Features

Introduction

MacBERT is an improved BERT with novel MLM as correction pre - training task, which mitigates the discrepancy of pre - training and fine - tuning.

Instead of masking with [MASK] token, which never appears in the ﬁne - tuning stage, we propose to use similar words for the masking purpose. A similar word is obtained by using Synonyms toolkit (Wang and Hu, 2017), which is based on word2vec (Mikolov et al., 2013) similarity calculations. If an N - gram is selected to mask, we will ﬁnd similar words individually. In rare cases, when there is no similar word, we will degrade to use random word replacement.

Here is an example of our pre - training task.

Property	Details
Original Sentence	we use a language model to predict the probability of the next word.
MLM	we use a language [M] to [M] ##di ##ct the pro [M] ##bility of the next word .
Whole word masking	we use a language [M] to [M] [M] [M] the [M] [M] [M] of the next word .
N - gram masking	we use a [M] [M] to [M] [M] [M] the [M] [M] [M] [M] [M] next word .
MLM as correction	we use a text system to ca ##lc ##ulate the po ##si ##bility of the next word .

Except for the new pre - training task, we also incorporate the following techniques.

Whole Word Masking (WWM)
N - gram masking
Sentence - Order Prediction (SOP)

Note that our MacBERT can be directly replaced with the original BERT as there is no differences in the main neural architecture.

For more technical details, please check our paper: Revisiting Pre - trained Models for Chinese Natural Language Processing

📄 License

This project is licensed under the "apache - 2.0" license.

📚 Documentation

Citation

If you find our resource or paper is useful, please consider including the following citation in your paper.

https://arxiv.org/abs/2004.13922

@inproceedings{cui-etal-2020-revisiting,
    title = "Revisiting Pre - Trained Models for {C}hinese Natural Language Processing",
    author = "Cui, Yiming  and
      Che, Wanxiang  and
      Liu, Ting  and
      Qin, Bing  and
      Wang, Shijin  and
      Hu, Guoping",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.findings-emnlp.58",
    pages = "657--668",
}

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご