Codegen25 7b Instruct P

🚀 CodeGen2.5-7B-instruct

CodeGen2.5: Small, but mighty, an autoregressive language model family for program synthesis

🚀 Quick Start

This model can be easily loaded using the AutoModelForCausalLM functionality.

Pre-requisite

Please install OpenAI tiktoken for the tokenizer.

pip install tiktoken==0.4.0

Causal sampling (code autocompletion)

For regular causal sampling, simply generate completions given the context:

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Salesforce/codegen25-7b-instruct", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Salesforce/codegen25-7b-instruct")

text = "def hello_world():"
input_ids = tokenizer(text, return_tensors="pt").input_ids
generated_ids = model.generate(input_ids, max_length=128)
print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))

Infill sampling

For infill sampling, we follow the CodeGen2 format:

• <mask_N>: N-th span to be masked. In practice, use <mask_1> to where you want to sample infill.
• <sep>: Separator token between the suffix and the infilled sample. See below.
• <eom>: "End-Of-Mask" token that model will output at the end of infilling. You may use this token to truncate the output.

For example, if we want to generate infill for the following cursor position of a function:

def hello_world():
    |
    return name

we construct an input to the model by

1. Inserting <mask_1> token in place of cursor position
2. Append <sep> token to indicate the boundary
3. Insert another <mask_1> to indicate which mask we want to infill.

The final snippet looks as follows:

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Salesforce/codegen25-7b-instruct", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Salesforce/codegen25-7b-instruct")


def format(prefix, suffix):
  return prefix + "<mask_1>" + suffix + "<|endoftext|>" + "<sep>" + "<mask_1>"


prefix = "def hello_world():\n    "
suffix = "    return name"
text = format(prefix, suffix)
input_ids = tokenizer(text, return_tensors="pt").input_ids
generated_ids = model.generate(input_ids, max_length=128)
print(tokenizer.decode(generated_ids[0], skip_special_tokens=False)[len(text):])

You might want to truncate the model output with <eom>.

✨ Features

• Program Synthesis: Capable of generating executable code given English prompts in the form of comment strings and can complete partially - generated code.
• Infill Capability: Similar to CodeGen2, it can perform infill operations.
• Multi - language Support: Supports multiple programming languages.

📦 Installation

Please install OpenAI tiktoken for the tokenizer.

pip install tiktoken==0.4.0

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Salesforce/codegen25-7b-instruct", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Salesforce/codegen25-7b-instruct")

text = "def hello_world():"
input_ids = tokenizer(text, return_tensors="pt").input_ids
generated_ids = model.generate(input_ids, max_length=128)
print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))

Advanced Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Salesforce/codegen25-7b-instruct", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Salesforce/codegen25-7b-instruct")


def format(prefix, suffix):
  return prefix + "<mask_1>" + suffix + "<|endoftext|>" + "<sep>" + "<mask_1>"


prefix = "def hello_world():\n    "
suffix = "    return name"
text = format(prefix, suffix)
input_ids = tokenizer(text, return_tensors="pt").input_ids
generated_ids = model.generate(input_ids, max_length=128)
print(tokenizer.decode(generated_ids[0], skip_special_tokens=False)[len(text):])

📚 Documentation

Model description

CodeGen2.5 is a family of autoregressive language models for program synthesis.

Building upon CodeGen2, the model is trained on StarCoderData for 1.4T tokens, achieving competitive results compared to StarCoderBase - 15.5B with less than half the size.

Like CodeGen2, this model is capable of infilling, and supports multiple programming languages.

We then further train on Python, then on instruction data. We release all the models as follows:

• CodeGen2.5-7B-multi: Trained on StarCoderData. Licensed under Apache - 2.0.
• CodeGen2.5-7B-mono: Further trained on additional Python tokens. Licensed under Apache - 2.0.
• CodeGen2.5-7B-instruct (this repo): Further trained from CodeGen2.5-7B-mono on instruction data. Research purposes only.

Evaluation results

We evaluate our models on HumanEval and HumanEval - Infill. Please refer to the blog for more details.

Intended use and limitations

As an autoregressive language model, CodeGen2.5 is capable of extracting features from given natural language and programming language texts, and calculating the likelihood of them. However, the model is intended for and best at program synthesis, that is, generating executable code given English prompts, where the prompts should be in the form of a comment string. The model can complete partially - generated code as well.

Attribution & Other Requirements

The pretraining dataset of the model was filtered for permissive licenses only. Nevertheless, the model can generate source code verbatim from the dataset. The code's license might require attribution and/or other specific requirements that must be respected. The data provider BigCode provides a search index that lets you search through the pretraining data to identify where generated code came from and apply the proper attribution to your code.

Ethical Considerations

This release is for research purposes only in support of an academic paper. Our models, datasets, and code are not specifically designed or evaluated for all downstream purposes. We strongly recommend users evaluate and address potential concerns related to accuracy, safety, and fairness before deploying this model. We encourage users to consider the common limitations of AI, comply with applicable laws, and leverage best practices when selecting use cases, particularly for high - risk scenarios where errors or misuse could significantly impact people’s lives, rights, or safety. For further guidance on use cases, refer to our AUP and AI AUP.

🔧 Technical Details

The model is built upon CodeGen2 and trained on StarCoderData for 1.4T tokens. It is further trained on Python and instruction data. It can achieve competitive results compared to StarCoderBase - 15.5B with less than half the size.

📄 License

The model has a license of "other". The specific models are licensed as follows:

• CodeGen2.5-7B-multi: Licensed under Apache - 2.0.
• CodeGen2.5-7B-mono: Licensed under Apache - 2.0.

BibTeX entry and citation info

Please cite CodeGen2 paper:

@article{Nijkamp2023codegen2,
  title={CodeGen2: Lessons for Training LLMs on Programming and Natural Languages},
  author={Nijkamp, Erik and Hayashi, Hiroaki and Xiong, Caiming and Savarese, Silvio and Zhou, Yingbo},
  journal={arXiv preprint},
  year={2023}
}