Codebert Base

🚀 CodeBERT-base

Pretrained weights for programming and natural languages integration, based on the CodeBERT model.

🚀 Quick Start

This README provides information about the pretrained weights of CodeBERT: A Pre-Trained Model for Programming and Natural Languages.

✨ Features

• Pretrained on bi-modal data of documents and code.
• Initialized with Roberta-base and trained with MLM+RTD objective.

📦 Installation

No specific installation steps are provided in the original document.

📚 Documentation

Training Data

Training Objective

This model is initialized with Roberta-base and trained with MLM+RTD objective (cf. the paper).

Usage

Please see the official repository for scripts that support "code search" and "code-to-document generation".

Reference

1. CodeBERT trained with Masked LM objective (suitable for code completion)
2. 🤗 Hugging Face's CodeBERTa (small size, 6 layers)

Citation

@misc{feng2020codebert,
    title={CodeBERT: A Pre-Trained Model for Programming and Natural Languages},
    author={Zhangyin Feng and Daya Guo and Duyu Tang and Nan Duan and Xiaocheng Feng and Ming Gong and Linjun Shou and Bing Qin and Ting Liu and Daxin Jiang and Ming Zhou},
    year={2020},
    eprint={2002.08155},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}