Codebertapy

🚀 CodeBERTaPy

CodeBERTaPy is a RoBERTa-like model trained on the CodeSearchNet dataset from GitHub for python by Manuel Romero. It offers efficient encoding of code corpora and can be used for masked language modeling prediction in Python code.

🚀 Quick Start

✨ Main Features

• Tokenizer: A Byte-level BPE tokenizer trained on the corpus using Hugging Face tokenizers. It encodes the code corpus efficiently, with sequences 33% to 50% shorter compared to the same corpus tokenized by gpt2/roberta.
• Model: A 6-layer, 84M parameters, RoBERTa-like Transformer model, similar to DistilBERT in terms of layers and heads. It's initialized from default settings and trained from scratch on the full python corpus for 4 epochs.

📦 Installation

The installation steps are not provided in the original README. If you need to use this model, you can install the transformers library which is used in the examples. You can install it via the following command:

pip install transformers

💻 Usage Examples

🔍 Basic Usage - Masked Language Modeling Prediction

PYTHON_CODE = """
fruits = ['apples', 'bananas', 'oranges']
for idx, <mask> in enumerate(fruits):
  print("index is %d and value is %s" % (idx, val))
""".lstrip()

🤔 Does the model know how to complete simple Python code?

from transformers import pipeline

fill_mask = pipeline(
    "fill-mask",
    model="mrm8488/CodeBERTaPy",
    tokenizer="mrm8488/CodeBERTaPy"
)

fill_mask(PYTHON_CODE)

## Top 5 predictions:

'val' # prob  0.980728805065155
'value'
'idx'
',val'
'_'

🎉 Yes! That was easy. Let's try with another Flask like example

PYTHON_CODE2 = """
@app.route('/<name>')
def hello_name(name):
    return "Hello {}!".format(<mask>)

if __name__ == '__main__':
    app.run()
""".lstrip()


fill_mask(PYTHON_CODE2)

## Top 5 predictions:

'name' # prob  0.9961813688278198
' name'
'url'
'description'
'self'

🎉 Yeah! It works. Let's try with another Tensorflow/Keras like example

PYTHON_CODE3="""
model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.<mask>(128, activation='relu'),
    keras.layers.Dense(10, activation='softmax')
])
""".lstrip()


fill_mask(PYTHON_CODE3)

## Top 5 predictions:

'Dense' # prob   0.4482928514480591
'relu'
'Flatten'
'Activation'
'Conv'

📚 Documentation

This work is heavily inspired on CodeBERTa by the Hugging Face team.

📄 License

The license information is not provided in the original README.

📚 CodeSearchNet Citation

@article{husain_codesearchnet_2019,
	title = {{CodeSearchNet} {Challenge}: {Evaluating} the {State} of {Semantic} {Code} {Search}},
	shorttitle = {{CodeSearchNet} {Challenge}},
	url = {http://arxiv.org/abs/1909.09436},
	urldate = {2020-03-12},
	journal = {arXiv:1909.09436 [cs, stat]},
	author = {Husain, Hamel and Wu, Ho-Hsiang and Gazit, Tiferet and Allamanis, Miltiadis and Brockschmidt, Marc},
	month = sep,
	year = {2019},
	note = {arXiv: 1909.09436},
}

Created by Manuel Romero/@mrm8488

Made with ♥ in Spain