Cmd Gen Codellama 13b Calm Demo

🚀 Command Generator Model Card

This is a Dialogue Understanding (DU) model developed by Rasa. It can empower assistants built with the Conversational AI with Language Models (CALM) approach developed by Rasa, offering efficient dialogue command generation.

✨ Features

• Input and Output: Takes the transcript of an ongoing conversation between an AI assistant and a user, along with structured information about the assistant's business logic as input. Outputs a short sequence of commands (usually 1 - 3) from a predefined list.
• Specific Command Generation: Can only generate commands to be interpreted by Rasa, not arbitrary text. Translates user messages into an internal grammar to progress the CALM conversation.

📦 Installation

No installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

# User message
I want to transfer money
# Generated command
StartFlow(transfer_money)

# User message
I want to transfer $55 to John
# Generated command
StartFlow(transfer_money), SetSlot(recipient, John), SetSlot(amount, 55)

📚 Documentation

Model Details

Model Description

This model takes the transcript of an ongoing conversation between an AI assistant and a user, as well as structured information about the assistant's business logic. As output, it produces a short sequence of commands (typically 1 - 3) from the following list:

• StartFlow(flow_name)
• SetSlot(slot_name, slot_value)
• CorrectSlot(slot_name, slot_value)
• Clarify(flow_name_1, flow_name_2, ...)
• ChitChat
• KnowledgeAnswer
• HumanHandoff
• Error

Note that this model can only produce commands to be interpreted by Rasa. It cannot be used to generate arbitrary text. The Command Generator translates user messages into this internal grammar, allowing CALM to progress the conversation.

Uses

Direct Use

This model can be directly used as part of the command generator component if the flows of your CALM assistant are similar to flows used in the rasa - calm - demo assistant.

Downstream Use [optional]

The model can also be used as a base model to fine - tune further on your own assistant's data using the fine - tuning recipe feature available in rasa pro.

Out - of - Scope Use

Since the model has been explicitly fine - tuned to output the grammar of commands, it shouldn't be used to generate any other free form content.

Bias, Risks, and Limitations

The Command Generator interprets conversations and translates user messages into commands. These commands are processed by Rasa to advance the conversations. This model does not generate text to be sent to an end user, and is incapable of generating problematic or harmful text. However, as with any pre - trained model, its predictions are susceptible to bias. For example, the accuracy of the model varies with the language used. The authors have tested the performance on English but haven't tried the model in any other language.

Training Details

Training Data

Trained on the train split of rasa/command - generation - calm - demo - v1.

Training Procedure

Trained using the notebook available here. Used a single A100 GPU with 80GB VRAM.

Evaluation

Testing Data, Factors & Metrics

Testing Data

Evaluated on the test split of rasa/command - generation - calm - demo - v1.

Metrics

F1 score per command type (StartFlow, SetSlot, etc.) is the main metric chosen to evaluate the model on the test split. This helps us understand which commands the model has learnt well and the ones that the model needs more training on.

Results

📄 License

This model is licensed under the Apache 2.0 license.