Code Trans T5 Small Api Generation Multitask

🚀 CodeTrans model for api recommendation generation

A pre-trained model for API recommendation generation using the T5 small model architecture, initially released in this repository.

🚀 Quick Start

This pre-trained model is designed for API recommendation generation, leveraging the T5 small model architecture. It was first introduced in this repository.

✨ Features

• Based on the t5-small model with its own SentencePiece vocabulary model.
• Utilized multi-task training on 13 supervised tasks in software development and 7 unsupervised datasets.

📦 Installation

No specific installation steps are provided in the original README.

💻 Usage Examples

Basic Usage

Here is how to use this model to generate Java function documentation using Transformers SummarizationPipeline:

from transformers import AutoTokenizer, AutoModelWithLMHead, SummarizationPipeline

pipeline = SummarizationPipeline(
    model=AutoModelWithLMHead.from_pretrained("SEBIS/code_trans_t5_small_api_generation_multitask"),
    tokenizer=AutoTokenizer.from_pretrained("SEBIS/code_trans_t5_small_api_generation_multitask", skip_special_tokens=True),
    device=0
)

tokenized_code = "parse the uses licence node of this package , if any , and returns the license definition if theres"
pipeline([tokenized_code])

Run this example in colab notebook.

📚 Documentation

Model description

This CodeTrans model is based on the t5-small model. It has its own SentencePiece vocabulary model. It used multi-task training on 13 supervised tasks in the software development domain and 7 unsupervised datasets.

Intended uses & limitations

The model could be used to generate API usage for Java programming tasks.

🔧 Technical Details

Training data

The supervised training tasks datasets can be downloaded on Link

Training procedure

Multi-task Pretraining

The model was trained on a single TPU Pod V3 - 8 for 500,000 steps in total, using sequence length 512 (batch size 4096). It has a total of approximately 220M parameters and was trained using the encoder - decoder architecture. The optimizer used is AdaFactor with inverse square root learning rate schedule for pre - training.

Evaluation results

For the code documentation tasks, different models achieve the following results on different programming languages (in BLEU score):

Test results:

Created by Ahmed Elnaggar | LinkedIn and Wei Ding | LinkedIn