Octocoder

🚀 OctoCoder

OctoCoder is an instruction - tuned model that can handle various programming language tasks. It is created by finetuning StarCoder on specific datasets, aiming to provide high - quality code generation and instruction - following capabilities.

🚀 Quick Start

Intended use

The model follows instructions provided in the input. You should always preface your input with "Question: " and finish it with "Answer:", for example: "Question: Please write a function in Python that performs bubble sort.\n\nAnswer:"

Feel free to share your generations in the Community tab!

Generation

# pip install -q transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "bigcode/octocoder"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)

inputs = tokenizer.encode("Question: Please write a function in Python that performs bubble sort.\n\nAnswer:", return_tensors="pt").to(device)
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

✨ Features

• Multilingual Support: Capable of handling over 80 programming languages.
• Instruction - Following: Can accurately follow instructions provided in the input for code generation.

📦 Installation

Although not explicitly provided in the original, if you want to use the model for generation, you need to install the transformers library:

pip install -q transformers

📚 Documentation

Model Summary

• Repository: [bigcode - project/octopack](https://github.com/bigcode - project/octopack)
• Paper: OctoPack: Instruction Tuning Code Large Language Models
• Languages: 80+ Programming languages

OctoPack🐙🎒

Training

Model

• Architecture: GPT - 2 model with multi - query attention and Fill - in - the - Middle objective
• Steps: 250k pretraining & 30 instruction tuning
• Pretraining tokens: 1 trillion pretraining & 2M instruction tuning
• Precision: bfloat16

Hardware

• Pretraining:
  • GPUs: 512 Tesla A100
  • Training time: 24 days
• Instruction tuning:
  • GPUs: 8 Tesla A100
  • Training time: 4 hours

Software

• Orchestration: [Megatron - LM/Transformers](https://github.com/bigcode - project/octopack#training)
• Neural networks: PyTorch

Results

📄 License

The model is released under the bigcode - openrail - m license.

📖 Citation

@article{muennighoff2023octopack,
      title={OctoPack: Instruction Tuning Code Large Language Models}, 
      author={Niklas Muennighoff and Qian Liu and Armel Zebaze and Qinkai Zheng and Binyuan Hui and Terry Yue Zhuo and Swayam Singh and Xiangru Tang and Leandro von Werra and Shayne Longpre},
      journal={arXiv preprint arXiv:2308.07124},
      year={2023}
}