Code Trans T5 Small Source Code Summarization Python Multitask Finetune

🚀 CodeTrans model for source code summarization in Python

A pre - trained model for Python source code summarization using the T5 small model architecture.

This model is a pre - trained one on the Python programming language, leveraging the t5-small model architecture. It was initially released in this repository. Trained on tokenized Python code functions, it performs optimally with such tokenized functions.

🚀 Quick Start

The CodeTrans model can be used to generate descriptions for Python functions or be fine - tuned for other Python code - related tasks. It can handle unparsed and untokenized Python code, but tokenized code generally yields better performance.

Here is an example of using this model to generate Python function documentation with the Transformers SummarizationPipeline:

from transformers import AutoTokenizer, AutoModelWithLMHead, SummarizationPipeline

pipeline = SummarizationPipeline(
    model=AutoModelWithLMHead.from_pretrained("SEBIS/code_trans_t5_small_source_code_summarization_python_multitask_finetune"),
    tokenizer=AutoTokenizer.from_pretrained("SEBIS/code_trans_t5_small_source_code_summarization_python_multitask_finetune", skip_special_tokens=True),
    device=0
)

tokenized_code = '''with open ( CODE_STRING , CODE_STRING ) as in_file : buf = in_file . readlines ( )  with open ( CODE_STRING , CODE_STRING ) as out_file : for line in buf :          if line ==   " ; Include this text   " :              line = line +   " Include below  "          out_file . write ( line ) '''
pipeline([tokenized_code])

You can run this example in colab notebook.

✨ Features

Model description

This CodeTrans model is built upon the t5-small model and comes with its own SentencePiece vocabulary model. It underwent multi - task training on 13 supervised tasks in the software development domain and 7 unsupervised datasets. Subsequently, it was fine - tuned for the source code summarization task of Python code snippets.

Intended uses & limitations

The model can generate descriptions for Python functions or be fine - tuned for other Python code tasks. It can work on unparsed and untokenized Python code, though tokenized code will lead to better performance.

📦 Installation

No installation steps were provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModelWithLMHead, SummarizationPipeline

pipeline = SummarizationPipeline(
    model=AutoModelWithLMHead.from_pretrained("SEBIS/code_trans_t5_small_source_code_summarization_python_multitask_finetune"),
    tokenizer=AutoTokenizer.from_pretrained("SEBIS/code_trans_t5_small_source_code_summarization_python_multitask_finetune", skip_special_tokens=True),
    device=0
)

tokenized_code = '''with open ( CODE_STRING , CODE_STRING ) as in_file : buf = in_file . readlines ( )  with open ( CODE_STRING , CODE_STRING ) as out_file : for line in buf :          if line ==   " ; Include this text   " :              line = line +   " Include below  "          out_file . write ( line ) '''
pipeline([tokenized_code])

📚 Documentation

Training data

The supervised training tasks datasets can be downloaded from Link

Training procedure

Multi - task Pretraining

The model was trained on a single TPU Pod V3 - 8 for a total of 500,000 steps, using a sequence length of 512 (batch size 4096). It has approximately 220M parameters in total and was trained with the encoder - decoder architecture. The optimizer used was AdaFactor with an inverse square root learning rate schedule for pre - training.

Fine - tuning

This model was then fine - tuned on a single TPU Pod V2 - 8 for a total of 600 steps, using a sequence length of 512 (batch size 256), using only the dataset containing Python code.

Evaluation results

For the source code summarization tasks, different models achieve the following results on different programming languages (in BLEU score):

Test results:

📄 License

No license information was provided in the original document, so this section is skipped.

Created by Ahmed Elnaggar | LinkedIn and Wei Ding | LinkedIn