Gpt Neo 1.3B Apps All

🚀 GPT-Neo-1.3B-APPS-all

GPT-Neo-1.3B-APPS-all is a model fine-tuned on the APPS dataset, specialized in solving programming tasks.

🚀 Quick Start

Please refer to our new GitHub Wiki which documents our efforts in detail in creating the open source version of GitHub Copilot

✨ Features

GPT-Neo-1.3B-APPS-all is a GPT-Neo-1.3B finetuned on APPS dataset. This model is specialized to solve programming tasks.

📦 Installation

No installation steps provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:

from transformers import AutoModelForCausalLM, AutoTokenizer, FlaxAutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("flax-community/gpt-code-clippy-1.3B-apps-alldata")

tokenizer = AutoTokenizer.from_pretrained("flax-community/gpt-code-clippy-1.3B-apps-alldata")

prompt = """

A function to greet user. Given a user name it should say hello

def greet(name):

ANSWER:

""" 

input_ids = tokenizer(prompt, return_tensors='pt').input_ids.to(device)

start = input_ids.size(1)

out = model.generate(input_ids, do_sample=True, max_length=50, num_beams=2, 

                     early_stopping=True, eos_token_id=tokenizer.eos_token_id, )

print(tokenizer.decode(out[0][start:]))

📚 Documentation

Model Description

GPT-Neo-1.3B-APPS-all is a GPT-Neo-1.3B finetuned on APPS dataset. This model is specialized to solve programming tasks.

Training data

Training procedure

The training script used to train this model can be found here.

Training is done for 5 epochs using AdamW optimizer and leaner decay learning rate schedule with 800 warmup steps. To reproduce the training one can use this command with the above script:

python run_clm_apps.py \
    --output_dir ./gpt-neo-1.3B-apps \
    --model_name_or_path EleutherAI/gpt-neo-1.3B \
    --dataset_name ./apps.py \
    --dataset_config_name formatted \
    --do_train --do_eval \
    --block_size="1024" \
    --per_device_train_batch_size="3" \
    --per_device_eval_batch_size="3" \
    --preprocessing_num_workers="16" \
    --learning_rate="8e-5" \
    --warmup_steps="800" \
    --adam_beta1="0.9" \
    --adam_beta2="0.98" \
    --weight_decay="0.1" \
    --overwrite_output_dir \
    --num_train_epochs="5" \
    --logging_steps="50" \
    --eval_steps="2000" \
    --report_to="wandb" \
    --dtype="bfloat16" \
    --save_strategy epoch \
    --gradient_accumulation_steps 1 \
    --all_data true \

Intended Use and Limitations

The model is finetuned to solve programming problems given a text description and optional starter code.

Limitations and Biases

⚠️ Important Note

This model is intended to be used for research purposes and comes with no guarantees of quality of generated code. The paper "Evaluating Large Language Models Trained on Code" from OpenAI has a good discussion on what the impact of a large language model trained on code could be. Therefore, some parts of their discuss are highlighted here as it pertains to this dataset and models that may be trained from it. As well as some differences in views from the paper, particularly around legal implications.

1. Over-reliance: This model may generate plausible solutions that may appear correct, but are not necessarily the correct solution. Not properly evaluating the generated code may cause have negative consequences such as the introduction of bugs, or the introduction of security vulnerabilities. Therefore, it is important that users are aware of the limitations and potential negative consequences of using this language model.

2. Economic and labor market impacts: Large language models trained on large code datasets such as this one that are capable of generating high-quality code have the potential to automate part of the software development process. This may negatively impact software developers. However, as discussed in the paper, as shown in the Summary Report of software developers from O*NET OnLine, developers don't just write software.

3. Biases: The model is trained on data containing prompt questions formatted in specific way. The performance of the model can be worse if the prompt formatting is different from that used in APPS dataset.

GPT-CC is finetuned GPT-Neo and might have inhereted biases and limitations from it. See GPT-Neo model card for details.

Eval results

Coming soon...

📄 License

This project is licensed under the MIT license.