Codebert Base Mlm

🚀 CodeBERT-base-mlm

Pretrained weights for programming and natural languages, based on the CodeBERT model.

🚀 Quick Start

This repository provides the pretrained weights of CodeBERT: A Pre-Trained Model for Programming and Natural Languages. Below are details about the model.

✨ Features

• Training Data: The model is trained on the code corpus of CodeSearchNet.
• Training Objective: Initialized with Roberta-base, this model is trained with a simple MLM (Masked Language Model) objective.

💻 Usage Examples

Basic Usage

from transformers import RobertaTokenizer, RobertaForMaskedLM, pipeline

model = RobertaForMaskedLM.from_pretrained('microsoft/codebert-base-mlm')
tokenizer = RobertaTokenizer.from_pretrained('microsoft/codebert-base-mlm')

code_example = "if (x is not None) <mask> (x>1)"
fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)

outputs = fill_mask(code_example)
print(outputs)

Expected results:

{'sequence': '<s> if (x is not None) and (x>1)</s>', 'score': 0.6049249172210693, 'token': 8}
{'sequence': '<s> if (x is not None) or (x>1)</s>', 'score': 0.30680200457572937, 'token': 50}
{'sequence': '<s> if (x is not None) if (x>1)</s>', 'score': 0.02133703976869583, 'token': 114}
{'sequence': '<s> if (x is not None) then (x>1)</s>', 'score': 0.018607674166560173, 'token': 172}
{'sequence': '<s> if (x is not None) AND (x>1)</s>', 'score': 0.007619690150022507, 'token': 4248}

📚 Documentation

Reference

1. Bimodal CodeBERT trained with MLM+RTD objective (suitable for code search and document generation)
2. 🤗 Hugging Face's CodeBERTa (small size, 6 layers)

Citation

@misc{feng2020codebert,
    title={CodeBERT: A Pre-Trained Model for Programming and Natural Languages},
    author={Zhangyin Feng and Daya Guo and Duyu Tang and Nan Duan and Xiaocheng Feng and Ming Gong and Linjun Shou and Bing Qin and Ting Liu and Daxin Jiang and Ming Zhou},
    year={2020},
    eprint={2002.08155},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

📄 License

No license information is provided in the original README, so this section is skipped.

🔧 Technical Details

No specific technical details beyond the training data and objective are provided, so this section is skipped.