llama-2-coder-7b Open-source Code Generation Model - Free Support for Programming Assistive Tasks

Home

Llama 2 Coder 7b

Developed by mrm8488

A code generation model fine-tuned based on Llama-2 7b, specifically designed for programming assistance tasks

Large Language Model

Transformers

OtherOpen Source License:Apache-2.0 #Code Generation #Instruction Fine-tuning #Programming Assistant

Downloads 191

Release Time : 7/26/2023

Model Overview

Llama-2-coder-7b is a large language model fine-tuned on the CodeAlpaca 20k instruction dataset, implemented using QLoRA method and PEFT library, focusing on code generation and programming assistance tasks.

Model Features

Code-Specific Fine-tuning

Fine-tuned specifically for programming tasks based on the CodeAlpaca 20k instruction dataset

Efficient Training Method

Implemented using QLoRA parameter-efficient fine-tuning technique and PEFT library

Programming Assistance Capability

Capable of understanding programming instructions and generating corresponding code solutions

Model Capabilities

Code Generation

Code Completion

Code Explanation

Programming Problem Solving

Use Cases

Development Assistance

Code Snippet Generation

Generate code snippets based on natural language descriptions

Can generate code in multiple programming languages

Code Modification

Modify existing code based on instructions

Such as adding XML navigation bar in the example

Education

Programming Learning

Help beginners understand programming concepts

🚀 Llama 2 Coder 🦙👩‍💻

Llama-2 7b fine-tuned on the CodeAlpaca 20k instructions dataset using QLoRA with the PEFT library.

🚀 Quick Start

The Llama-2 Coder is a fine - tuned version of Llama-2 7b on the CodeAlpaca 20k instructions dataset using the QLoRA method with the PEFT library.

✨ Features

Fine - tuned on the CodeAlpaca 20k instructions dataset.
Utilizes the QLoRA method with the PEFT library.

📚 Documentation

🧠 Model description

Llama-2 is a collection of pretrained and fine - tuned generative text models with parameter scales ranging from 7 billion to 70 billion. Meta developed and publicly released the Llama 2 family of large language models (LLMs). Our fine - tuned LLMs, called Llama - 2 - Chat, are optimized for dialogue use cases. Llama - 2 - Chat models outperform open - source chat models on most benchmarks we tested, and in our human evaluations for helpfulness and safety, are on par with some popular closed - source models like ChatGPT and PaLM.

📚 Training and evaluation data

CodeAlpaca_20K: contains 20K instruction - following data used for fine - tuning the Code Alpaca model.

⚙ Training hyperparameters

    optim="paged_adamw_32bit",
    num_train_epochs = 2,
    eval_steps=50,
    save_steps=50,
    evaluation_strategy="steps",
    save_strategy="steps",
    save_total_limit=2,
    seed=66,
    load_best_model_at_end=True,
    logging_steps=1,
    learning_rate=2e-4,
    fp16=True,
    bf16=False,
    max_grad_norm=0.3,
    warmup_ratio=0.03,
    group_by_length=True,
    lr_scheduler_type="constant"

🗒️ Training results

Step	Training Loss	Validation Loss
50	0.624400	0.600070
100	0.634100	0.592757
150	0.545800	0.586652
200	0.572500	0.577525
250	0.528000	0.590118

📊 Eval results

WIP

💻 Usage Examples

👩‍💻 Basic Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig

model_id = "mrm8488/llama-2-coder-7b"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(model_id).to("cuda")

def create_prompt(instruction):
  system = "You are a coding assistant that will help the user to resolve the following instruction:"
  instruction = "### Instruction: " + instruction
  return system + "\n" + instruction + "\n\n" + "### Solution:" + "\n"

def generate(
        instruction,
        max_new_tokens=128,
        temperature=0.1,
        top_p=0.75,
        top_k=40,
        num_beams=4,
        **kwargs,
):
    prompt = create_prompt(instruction)
    print(prompt)
    inputs = tokenizer(prompt, return_tensors="pt")
    input_ids = inputs["input_ids"].to("cuda")
    attention_mask = inputs["attention_mask"].to("cuda")
    generation_config = GenerationConfig(
        temperature=temperature,
        top_p=top_p,
        top_k=top_k,
        num_beams=num_beams,
        **kwargs,
    )
    with torch.no_grad():
        generation_output = model.generate(
            input_ids=input_ids,
            attention_mask=attention_mask,
            generation_config=generation_config,
            return_dict_in_generate=True,
            output_scores=True,
            max_new_tokens=max_new_tokens,
            early_stopping=True
        )
    s = generation_output.sequences[0]
    output = tokenizer.decode(s)
    return output.split("### Solution:")[1].lstrip("\n")

instruction = """
Edit the following XML code to add a navigation bar to the top of a web page
<html>
<head>
  <title>CliBrAIn</title>
</head>
"""
print(generate(instruction))

📄 License

This project is licensed under the apache - 2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご