Orca Mini 13b

🚀 orca_mini_13b

An OpenLLaMa-13B model trained on explain tuned datasets, leveraging instructions from multiple sources and Orca Research Paper approaches.

🚀 Quick Start

This orca_mini_13b is an OpenLLaMa-13B model trained on explain tuned datasets. It was created using instructions and inputs from WizardLM, Alpaca, and Dolly-V2 datasets, applying the dataset construction approaches from the Orca Research Paper.

✨ Features

• Trained on explain tuned datasets created from WizardLM, Alpaca, and Dolly-V2, using Orca Research Paper approaches.
• Helps the model learn the thought process from the teacher model (ChatGPT - gpt-3.5-turbo-0301 version).

📦 Installation

The README does not provide specific installation steps, so this section is skipped.

💻 Usage Examples

Basic Usage

import torch
from transformers import LlamaForCausalLM, LlamaTokenizer

# Hugging Face model_path
model_path = 'psmathur/orca_mini_13b'
tokenizer = LlamaTokenizer.from_pretrained(model_path)
model = LlamaForCausalLM.from_pretrained(
    model_path, torch_dtype=torch.float16, device_map='auto',
)


#generate text function
def generate_text(system, instruction, input=None):
    
    if input:
        prompt = f"### System:\n{system}\n\n### User:\n{instruction}\n\n### Input:\n{input}\n\n### Response:\n"
    else:
        prompt = f"### System:\n{system}\n\n### User:\n{instruction}\n\n### Response:\n"
    
    tokens = tokenizer.encode(prompt)
    tokens = torch.LongTensor(tokens).unsqueeze(0)
    tokens = tokens.to('cuda')

    instance = {'input_ids': tokens,'top_p': 1.0, 'temperature':0.7, 'generate_len': 1024, 'top_k': 50}

    length = len(tokens[0])
    with torch.no_grad():
        rest = model.generate(
            input_ids=tokens, 
            max_length=length+instance['generate_len'], 
            use_cache=True, 
            do_sample=True, 
            top_p=instance['top_p'],
            temperature=instance['temperature'],
            top_k=instance['top_k']
        )    
    output = rest[0][length:]
    string = tokenizer.decode(output, skip_special_tokens=True)
    return f'[!] Response: {string}'

# Sample Test Instruction Used by Youtuber Sam Witteveen https://www.youtube.com/@samwitteveenai
system = 'You are an AI assistant that follows instruction extremely well. Help as much as you can.'
instruction = 'Write a letter to Sam Altman, CEO of OpenAI, requesting him to convert GPT4 a private model by OpenAI to an open source project'
print(generate_text(system, instruction))

Advanced Usage

The README does not provide advanced usage examples, so this part is not added.

[!] Response:
Dear Sam Altman,

I am writing to request that you convert the GPT4 private model developed by OpenAI to an open source project. As a user of OpenAI, I have been waiting for the day when I can use the advanced natural language processing capabilities of GPT4 in a more open and accessible way.

While OpenAI has made significant progress in developing AI applications, it has primarily focused on building private models that are not accessible to the general public. However, with the recent release of GPT-3, there is a growing demand for more open and accessible AI tools.

Converting GPT4 to an open source project would allow for greater transparency, collaboration, and innovation. It would also help to build trust in the technology and ensure that it is used ethically and responsibly.

I urge you to consider converting GPT4 to an open source project. This would be a significant contribution to the AI community and would help to create a more open and accessible future.

Thank you for your consideration.

Sincerely,

[Your Name]

📚 Documentation

Dataset

We build explain tuned WizardLM dataset ~70K, Alpaca dataset ~52K, and Dolly-V2 dataset ~15K using approaches from the Orca Research Paper.

We leverage all 15 system instructions provided in the Orca Research Paper to generate custom datasets, different from the vanilla instruction tuning approaches used by the original datasets. This helps the student model (this model) learn the thought process from the teacher model, which is ChatGPT (gpt-3.5-turbo-0301 version).

Training

The training configurations are provided in the table below. The training was conducted on 8x A100(80G) GPUs for around 15 hours at a cost of $180 using Lambda Labs.

We used DeepSpeed with fully sharded data parallelism, also known as ZeRO stage 3, by writing our own fine-tuning scripts and leveraging some of the model training code provided by the amazing OpenAlpaca repo.

Next Goals

1. Try more data like actually using FLAN-v2, just like the Orca Research Paper (open for suggestions).
2. Provide more options for Text generation UI (e.g., https://github.com/oobabooga/text-generation-webui).
3. Provide 4bit GGML/GPTQ quantized model (possibly with help from TheBloke).

Limitations & Biases

This model can produce factually incorrect output and should not be relied on to produce factually accurate information. It was trained on various public datasets. Despite great efforts to clean the pretraining data, it is possible that this model could generate lewd, biased, or otherwise offensive outputs.

Disclaimer

The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.

Citation

If you found wizardlm_alpaca_dolly_orca_open_llama_13b useful in your research or applications, please kindly cite using the following BibTeX:

@misc{orca_mini_13b,
  author = {Pankaj Mathur},
  title = {orca_mini_13b: An explain tuned OpenLLaMA-13b model on custom wizardlm, alpaca, & dolly datasets},
  year = {2023},
  publisher = {GitHub, HuggingFace},
  journal = {GitHub repository, HuggingFace repository},
  howpublished = {\url{https://github.com/pankajarm/wizardlm_alpaca_dolly_orca_open_llama_13b}, \url{https://https://huggingface.co/psmathur/wizardlm_alpaca_dolly_orca_open_llama_13b}},
}

@misc{mukherjee2023orca,
      title={Orca: Progressive Learning from Complex Explanation Traces of GPT-4}, 
      author={Subhabrata Mukherjee and Arindam Mitra and Ganesh Jawahar and Sahaj Agarwal and Hamid Palangi and Ahmed Awadallah},
      year={2023},
      eprint={2306.02707},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@software{openlm2023openllama,
  author = {Xinyang Geng and Hao Liu},
  title = {OpenLLaMA: An Open Reproduction of LLaMA},
  month = May,
  year = 2023,
  url = {https://github.com/openlm-research/open_llama}
}

@misc{openalpaca,
  author = {Yixuan Su and Tian Lan and Deng Cai},
  title = {OpenAlpaca: A Fully Open-Source Instruction-Following Model Based On OpenLLaMA},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/yxuansu/OpenAlpaca}},
}

@misc{alpaca,
  author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
  title = {Stanford Alpaca: An Instruction-following LLaMA model},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
}

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

📄 License

This model is licensed under cc-by-nc-sa-4.0.