Airoboros Gpt 3.5 Turbo 100k 7b

🚀 airoboros-gpt-3.5-turbo-100k-7b

This is a 7B parameter model fine - tuned on 100k synthetic instruction/response pairs. It offers high - quality performance in various NLP tasks and has a context length of 2048.

🚀 Quick Start

This repo is old and uses an old prompt format. You may want to look at airoboros-13b.

✨ Features

This is a 7b parameter model, fine - tuned on 100k synthetic instruction/response pairs generated by gpt - 3.5 - turbo using the version of self - instruct airoboros. The context length for this model is 2048.

Links

• airoboros
• instructions.jsonl
• topics.txt

📦 Installation

Prompt generation

airoboros generate-instructions --instruction-count 100000 --concurrency 100 --temperature 1.0

Fine - tuning

The instructions.jsonl file was converted to conversation style expected by the FastChat training scripts, and then trained with:

torchrun --nproc_per_node=8 --master_port=20001 train_mem.py \
        --model_name_or_path /workspace/llama-7b-hf \
        --data_path ./as_conversations.json \
        --bf16 True \
        --output_dir /workspace/airoboros-gpt-3.5-100k-7b \
        --num_train_epochs 3 \
        --per_device_train_batch_size 4 \
        --per_device_eval_batch_size 32 \
        --gradient_accumulation_steps 4 \
        --evaluation_strategy "steps" \
        --eval_steps 1500 \
        --save_strategy "steps" \
        --save_steps 1500 \
        --save_total_limit 8 \
        --learning_rate 2e-5 \
        --weight_decay 0. \
        --warmup_ratio 0.04 \
        --lr_scheduler_type "cosine" \
        --logging_steps 1 \
        --fsdp "full_shard auto_wrap offload" \
        --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
        --tf32 True \
        --model_max_length 2048 \
        --gradient_checkpointing True \
        --lazy_preprocess True

Training took roughly 22 hours on 8x nvidia A100 80GB.

Conversion to conversation style

import json
import uuid
inputs = [json.loads(line) for line in open("instructions.jsonl").readlines()]
conversations = []
for row in inputs:
    inputs = row['instruction']
    conversations.append({
        "id": str(uuid.uuid4()),
        "conversations": [
            {
                "from": "human",
                "value": inputs,
            },
            {
                "from": "gpt",
                "value": row['response']
            },
        ],
    })
with open("as_conversations.json", "w") as outfile:
    outfile.write(json.dumps(conversations, indent=2))

📚 Documentation

Evaluation

I used the same questions from WizardVicunaLM:

If we use gpt - 3.5 as the baseline (as wizardvicuna/vicuna did), we get the following scores:

Average scores:

gpt3.5                             1.000000
wizard - vicuna - 13b                  0.934090
vicuna - 13b                         0.900847
wizard - 7b                          0.902428
airoboros - gpt - 3.5 - turbo - 100k - 7b    0.926496

As you can see, the 7b airoboros model performs well, even compared to 13b models.

💻 Usage Examples

Basic Usage

The easiest way to use this model is via fastchat:

python -m fastchat.serve.cli --model-path ./airoboros-gpt-3.5-turbo-100k-7b --temperature 1.0

Advanced Usage

# For example:
Human: Write an email introducing a new instruction - tuned AI model named airoboros.  The key features about airoboros are: 1. it was fine - tuned with entirely machine - generated synthetic instruction/response pairs  2. the 7 billion parameter model performs very well in comparison to 13b models  3. the synthetic prompt generation tool is free, open - source, and highly customizable.
Assistant: Subject: Introducing airoboros - a new instruction - tuned AI model

Dear [Recipient],

I am excited to introduce a new instruction - tuned AI model named airoboros. This model was developed by our team with the aim of providing high - quality natural language processing capabilities at scale.

airoboros is a 7 billion parameter model that was fine - tuned with entirely machine - generated synthetic instruction/response pairs. This approach allowed us to achieve state - of - the - art performance in various NLP tasks such as question answering, sentiment analysis, and machine translation. In fact, when compared to other popular 13b models, airoboros performed better on several benchmarks.

Moreover, airoboros's synthetic prompt generation tool is free, open - source, and highly customizable. This means that developers can easily integrate our model into their applications without having to worry about licensing fees or restrictions.

We believe that airoboros will be an excellent addition to your workflow and hope you find it useful. If you have any questions or concerns, please do not hesitate to contact us.

Best regards,

[Your Name]

[Your Company Name]

⚠️ Important Note

Keep in mind, this is an instruction - tuned model and not well - suited to chat.

Usage and License Notices

All airoboros models and datasets are intended and licensed for research use only. I've used the 'cc - nc - 4.0' license, but really it is subject to a custom/special license because:

• the base model is LLaMa, which has its own special research license
• the dataset(s) were generated with OpenAI (gpt - 4 and/or gpt - 3.5 - turbo), which has a clause saying the data can't be used to create models to compete with openai

So, to reiterate: this model (and datasets) cannot be used commercially.

📄 License

This project uses the 'cc - nc - 4.0' license, but is subject to a custom/special license due to the base model and dataset generation details.