Erlangshen DeBERTa V2 320M Chinese

🚀 Erlangshen-DeBERTa-v2-320M-Chinese

A Chinese DeBERTa-v2-Large model with 320M parameters, excelling at NLU tasks and adopting Whole Word Masking.

• Main Page: Fengshenbang
• Github: Fengshenbang-LM

🚀 Quick Start

This is a Chinese DeBERTa-v2-Large model with 320 million parameters, which is good at handling NLU tasks and uses Whole Word Masking.

✨ Features

• NLU Expertise: Specialized in solving NLU tasks.
• Whole Word Masking: Adopted in the MLM process.
• Chinese Version: Tailored for the Chinese language.

📦 Model Taxonomy

📚 Documentation

Model Information

Reference paper: DeBERTa: Decoding-enhanced BERT with Disentangled Attention

To obtain a Chinese DeBERTa-v2-large (320M), we pre-trained the model using the WuDao Corpora (180 GB version). We employed Whole Word Masking (wwm) in MLM. Specifically, we used the fengshen framework for pre-training, which took approximately 7 days with 8 A100 (80G) GPUs.

Performance on Downstream Tasks

We present the results (dev set) on the following tasks:

💻 Usage Examples

Basic Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer, FillMaskPipeline
import torch

tokenizer=AutoTokenizer.from_pretrained('IDEA-CCNL/Erlangshen-DeBERTa-v2-320M-Chinese', use_fast=False)
model=AutoModelForMaskedLM.from_pretrained('IDEA-CCNL/Erlangshen-DeBERTa-v2-320M-Chinese')
text = '桂林是世界闻名的旅游城市,它有[MASK]江。'
fillmask_pipe = FillMaskPipeline(model, tokenizer, device=0)
print(fillmask_pipe(text, top_k=10))

📄 License

This project is licensed under the Apache-2.0 license.

📖 Citation

If you are using the resource for your work, please cite the our paper:

@article{fengshenbang,
  author    = {Jiaxing Zhang and Ruyi Gan and Junjie Wang and Yuxiang Zhang and Lin Zhang and Ping Yang and Xinyu Gao and Ziwei Wu and Xiaoqun Dong and Junqing He and Jianheng Zhuo and Qi Yang and Yongfeng Huang and Xiayu Li and Yanghan Wu and Junyu Lu and Xinyu Zhu and Weifeng Chen and Ting Han and Kunhao Pan and Rui Wang and Hao Wang and Xiaojun Wu and Zhongshen Zeng and Chongpei Chen},
  title     = {Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence},
  journal   = {CoRR},
  volume    = {abs/2209.02970},
  year      = {2022}
}

You can also cite our website:

@misc{Fengshenbang-LM,
  title={Fengshenbang-LM},
  author={IDEA-CCNL},
  year={2021},
  howpublished={\url{https://github.com/IDEA-CCNL/Fengshenbang-LM}},
}