Erlangshen DeBERTa V2 97M Chinese

🚀 Erlangshen-DeBERTa-v2-97M-Chinese

A Chinese DeBERTa-v2-Base model with 97M parameters, excelling in NLU tasks and adopting Whole Word Masking.

• Main Page: Fengshenbang
• Github: Fengshenbang-LM

🚀 Quick Start

This is a Chinese DeBERTa-v2-Base model with 97M parameters. It is good at handling NLU tasks and uses Whole Word Masking.

✨ Features

• Specialized in NLU tasks.
• Adopts Whole Word Masking.
• Chinese version of DeBERTa-v2-Base with 97M parameters.

📦 Installation

No installation steps provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer, FillMaskPipeline
import torch

tokenizer=AutoTokenizer.from_pretrained('IDEA-CCNL/Erlangshen-DeBERTa-v2-97M-Chinese', use_fast=False)
model=AutoModelForMaskedLM.from_pretrained('IDEA-CCNL/Erlangshen-DeBERTa-v2-97M-Chinese')
text = '生活的真谛是[MASK]。'
fillmask_pipe = FillMaskPipeline(model, tokenizer, device=7)
print(fillmask_pipe(text, top_k=10))

📚 Documentation

Model Taxonomy

Model Information

Reference paper: DeBERTa: Decoding-enhanced BERT with Disentangled Attention

To get a Chinese DeBERTa-v2-Base (97M), we use WuDao Corpora (180 GB version) for pre-training. We employ the Whole Word Masking (wwm) in MLM. Specifically, we use the fengshen framework in the pre-training phase which cost about 7 days with 24 A100 GPUs.

Performance on Downstream Tasks

We present the results (dev set) on the following tasks:

📄 License

This project is licensed under the Apache-2.0 license.

📖 Citation

If you are using the resource for your work, please cite the our paper:

@article{fengshenbang,
  author    = {Jiaxing Zhang and Ruyi Gan and Junjie Wang and Yuxiang Zhang and Lin Zhang and Ping Yang and Xinyu Gao and Ziwei Wu and Xiaoqun Dong and Junqing He and Jianheng Zhuo and Qi Yang and Yongfeng Huang and Xiayu Li and Yanghan Wu and Junyu Lu and Xinyu Zhu and Weifeng Chen and Ting Han and Kunhao Pan and Rui Wang and Hao Wang and Xiaojun Wu and Zhongshen Zeng and Chongpei Chen},
  title     = {Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence},
  journal   = {CoRR},
  volume    = {abs/2209.02970},
  year      = {2022}
}

You can also cite our website:

@misc{Fengshenbang-LM,
  title={Fengshenbang-LM},
  author={IDEA-CCNL},
  year={2021},
  howpublished={\url{https://github.com/IDEA-CCNL/Fengshenbang-LM}},
}