Llama 2 7b Chat Hf Function Calling V2

🚀 Function Calling Llama 2 + Yi + Mistral + Zephyr + Deepseek Coder Models (version 2)

Function Calling Llama 2 extends the Hugging Face Llama 2 models with function calling capabilities, enabling structured JSON responses.

✨ Features

• Function calling Llama extends the hugging face Llama 2 models with function calling capabilities.
• The model responds with a structured json argument with the function name and arguments.

📈 Recent Updates

• Nov 15th 2023: Added Yi 200k context models in 6B and 34B form.
• Nov 8th 2023: Added Zephyr beta, an improved version of Mistral 7B (achieved via DPO).
• November 6th 2023: Added Deepseek Coder 1.3B, 6.7B and 33B.
• October 11th 2023: Added Mistral 7B with function calling.
• October 11th 2023: New models pushed, trained on an improved underlying dataset.

💡 Improvements with v2

1. Shortened syntax: Only function descriptions are needed for inference and no added instruction is required.
2. Function descriptions outside system prompt: This avoids the behaviour of function calling being affected by how the system prompt had been trained to influence the model.

📦 Latest Models

• Yi-6B-200k context with function calling:
  • Base Model
  • PEFT Adapters - Paid, purchase here
• Yi-34B-200k context with function calling:
  • Base Model
  • PEFT Adapters
  • AWQ
  • [GGUF - files are in the main branch of the base model] - Paid, purchase here
• Deepseek-Coder-1.3B-Instruct with function calling:
  • Base Model
  • PEFT Adapters - Paid, purchase here
• Llama-7B-chat with function calling:
  • Base Model
  • PEFT Adapters
  • [GGUF - files are in the main branch of the base model] - Free
• zephyr-7b-beta with function calling:
  • Base Model
  • PEFT Adapters
  • [GGUF - files are in the main branch of the base model] - Paid, purchase here
• Mistral-7B-Instruct-v0.1 with function calling:
  • Base Model
  • PEFT Adapters - Paid, purchase here
• Deepseek-Coder-6.7B-Instruct with function calling:
  • Base Model
  • PEFT Adapters - Paid, purchase here
• Deepseek-Coder-33B-Instruct with function calling:
  • Base Model
  • PEFT Adapters - Paid, purchase here
• CodeLlama-34B-Instruct with function calling:
  • Base Model
  • PEFT Adapters - Paid, purchase here
• Llama-70B-chat with function calling:
  • Base Model
  • PEFT Adapters - Paid, purchase here

📦 Other Models

• Llama-13B-chat with function calling:
  • Base Model
  • PEFT Adapters - Paid, purchase here

❓ Which model is best for what?

1. Model size and function calling: Larger models are better at handling function calling. The cross entropy training losses are approximately 0.5 for 7B, 0.4 for 13B, 0.3 for 70B. The absolute numbers don't mean anything but the relative values offer a sense of relative performance.
2. Function descriptions: Provide very clear function descriptions, including whether the arguments are required or what the default values should be.
3. Post - processing: Make sure to post - process the language model's response to check that all necessary information is provided by the user. If not, prompt the user to let them know they need to provide more info (e.g. their name, order number etc.)

Check out this video overview of performance here

💡 Some short tips based on models as of November 2023

• DeepSeek Coder (all sizes): Best coding model.
• Yi 34B: Best for long context.
• Llama 70B: Strongest overall model (4k context).
• Mistral 7B: Best model if you have only 8 GB of VRAM (run with quantization).
• Zephyr: Better than Mistral 7B but is not openly licensed for commercial use.

📄 License

• Llama-7B with function calling: Licensed according to the Meta Community license.
• Mistral-7B, Llama-13B, Code-llama-34b, Llama-70B and Falcon-180B with function calling: Require the purchase of access.
  • Commercial license purchase required per user.
  • Licenses are not transferable to other users/entities.
• All Llama models with function calling: Further subject to terms in the Meta license.
• Yi models: Subject to the Yi license, which permits commercial use as of Nov 15th 2023.
• Zephr models: Generated using Ultrachat, which relies on openai. OpenAI does not permit the use of it's models to train competitive models. This makes it unclear as to whether Zephyr may be used commercial. Buyers/users do so at their sole risk.

📊 Dataset

The dataset used for training this model can be found at Trelis Function Calling Extended Dataset.

💻 Usage Examples

🚀 Quick Start in Google Colab

Try out this notebook fLlama_Inference notebook

📡 Text Generation Inference

You can use this model with text-generation-interface and chat-ui

Here is the github for setup

And here is a video showing it working with llama-2-7b-chat-hf-function-calling-v2 (note that we've now moved to v2)

Note that you'll still need to code the server - side handling of making the function calls (which obviously depends on what functions you want to use).

⚡ Runpod Quickstart

For a quickstart with runpod, you can use this template: here

Once up and running, you can make queries to:

https://{YOUR_POD_ID}-8080.proxy.runpod.net

Then, you can make queries to the api as follows:

curl https://{YOUR_POD_ID}-8080.proxy.runpod.net/generate \
    -X POST \
    -d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":20}}' \
    -H 'Content-Type: application/json'

Or use /generate_stream for streaming. You can also write python scripts and use python to make requests. More info from the text - generation - inference github repo

💻 Run on your laptop

Run on your laptop video and juypter notebook

After running llama.cpp server, you can call the server with this command, with thanks to @jdo300:

import requests
import json

# Define the roles and markers
B_FUNC, E_FUNC = "<FUNCTIONS>", "</FUNCTIONS>\n\n"
B_INST, E_INST = "[INST] ", " [/INST]" #Llama style
# B_INST, E_INST = "\n### Instruction:\n", "\n### Response:\n" #DeepSeek Coder Style
# B_INST, E_INST = "Human: ", " Assistant: " #Yi Style

# Define the function metadata
function_metadata = {
    "function": "search_bing",
    "description": "Search the web for content on Bing. This allows users to search online/the internet/the web for content.",
    "arguments": [
        {
            "name": "query",
            "type": "string",
            "description": "The search query string"
        }
    ]
}

# Define the user prompt
user_prompt = 'Search for the latest news on AI.'

# Format the function list and prompt
function_list = json.dumps(function_metadata, indent=4)
prompt = f"{B_FUNC}{function_list.strip()}{E_FUNC}{B_INST}{user_prompt.strip()}{E_INST}\n\n"

# Define the API endpoint
url = "http:/localhost:8080/completion"

# Send the POST request to the API server
response = requests.post(url, json={"prompt": prompt})

# Print the response
print(response.json())

📚 Documentation

⚙️ Syntax - Prompt Templates

The function descriptions must be wrapped within a function block. You can put this function below before or after the system message block.

Example without a system message:

  # Define the roles and markers
  B_FUNC, E_FUNC = "<FUNCTIONS>", "</FUNCTIONS>\n\n"
  B_INST, E_INST = "[INST] ", " [/INST]" #Llama style
  # B_INST, E_INST = "\n### Instruction:\n", "\n### Response:\n" #DeepSeek Coder Style
  # B_INST, E_INST = "Human: ", " Assistant: " #Yi Style

  functionList = {function_1_metadata}{function_2_metadata}...
  user_prompt = '...'

  # Format your prompt template
  prompt = f"{B_FUNC}{functionList.strip()}{E_FUNC}{B_INST}{user_prompt.strip()}{E_INST}\n\n"

Example with a system message:

  # Define the roles and markers
  B_FUNC, E_FUNC = "<FUNCTIONS>", "</FUNCTIONS>\n\n"
  B_INST, E_INST = "[INST] ", " [/INST]" #Llama style
  # B_INST, E_INST = "\n### Instruction:\n", "\n### Response:\n" #DeepSeek Coder Style
  # B_INST, E_INST = "Human: ", " Assistant: " #Yi Style
  B_SYS, E_SYS = "<<SYS>>\n", "\n<</SYS>>\n\n"

  # assuming functionList is defined as above
  system_prompt = '...'
  user_prompt = '...'

  # Format your prompt template
  prompt = f"{B_FUNC}{functionList.strip()}{E_FUNC}{B_INST}{B_SYS}{system_prompt.strip()}{E_SYS}{user_prompt.strip()}{E_INST}\n\n"

Notice that the function block is placed at the very start of the sequence, before 'B_INST'.

⚙️ Function Metadata Template

functionMetadata should be a string representation of a JSON object, like this:

"functionMetadata": {
        "function": "search_bing",
        "description": "Search the web for content on Bing. This allows users to search online/the internet/the web for content.",
        "arguments": [
            {
                "name": "query",
                "type": "string",
                "description": "The search query string"
            }
        ]
    }

and the language model should respond with a json object formatted like this:

{
    "function": "function_name",
    "arguments": {
        "argument1": "argument_value",
        "argument2": "argument_value"
    }
}

It is recommended to handle cases where:

• There is no json object in the response
• The response contains text in addition to the json response

⚙️ Sample functionList

{
    "function": "search_bing",
    "description": "Search the web for content on Bing. This allows users to search online/the internet/the web for content.",
    "arguments": [
        {
            "name": "query",
            "type": "string",
            "description": "The search query string"
        }
    ]
}

{
    "function": "search_arxiv",
    "description": "Search for research papers on ArXiv. Make use of AND, OR and NOT operators as appropriate to join terms within the query.",
    "arguments": [
        {
            "name": "query",
            "type": "string",
            "description": "The search query string"
        }
    ]
}

⚙️ Training Set Argument Types

Models were fine - tuned on argument types including strings, numbers and arrays. The training set includes function calls with 0, 1, 2 or 3 arguments.