đ orca_mini_3b
An OpenLLaMa-3B model trained on explain tuned datasets, created using Instructions and Input from WizardLM, Alpaca & Dolly-V2 datasets and applying Orca Research Paper dataset construction approaches.
đ Quick Start
Use orca-mini-3b for Free on Google Colab with T4 GPU :)
⨠Features
An OpenLLaMa-3B model trained on explain tuned datasets. It uses instructions and input from WizardLM, Alpaca & Dolly-V2 datasets and applies the dataset construction approaches from the Orca Research Paper.
đĻ Installation
No specific installation steps are provided in the original README, so this section is skipped.
đģ Usage Examples
Basic Usage
import torch
from transformers import LlamaForCausalLM, LlamaTokenizer
model_path = 'psmathur/orca_mini_3b'
tokenizer = LlamaTokenizer.from_pretrained(model_path)
model = LlamaForCausalLM.from_pretrained(
model_path, torch_dtype=torch.float16, device_map='auto',
)
def generate_text(system, instruction, input=None):
if input:
prompt = f"### System:\n{system}\n\n### User:\n{instruction}\n\n### Input:\n{input}\n\n### Response:\n"
else:
prompt = f"### System:\n{system}\n\n### User:\n{instruction}\n\n### Response:\n"
tokens = tokenizer.encode(prompt)
tokens = torch.LongTensor(tokens).unsqueeze(0)
tokens = tokens.to('cuda')
instance = {'input_ids': tokens,'top_p': 1.0, 'temperature':0.7, 'generate_len': 1024, 'top_k': 50}
length = len(tokens[0])
with torch.no_grad():
rest = model.generate(
input_ids=tokens,
max_length=length+instance['generate_len'],
use_cache=True,
do_sample=True,
top_p=instance['top_p'],
temperature=instance['temperature'],
top_k=instance['top_k']
)
output = rest[0][length:]
string = tokenizer.decode(output, skip_special_tokens=True)
return f'[!] Response: {string}'
system = 'You are an AI assistant that follows instruction extremely well. Help as much as you can.'
instruction = 'Write a letter to Sam Altman, CEO of OpenAI, requesting him to convert GPT4 a private model by OpenAI to an open source project'
print(generate_text(system, instruction))
Advanced Usage
đ Documentation
Dataset
We build explain tuned WizardLM dataset ~70K, Alpaca dataset ~52K & Dolly-V2 dataset ~15K created using approaches from Orca Research Paper.
We leverage all of the 15 system instructions provided in Orca Research Paper to generate custom datasets, in contrast to vanilla instruction tuning approaches used by original datasets.
This helps the student model (this model) to learn the thought process from the teacher model, which is ChatGPT (gpt-3.5-turbo-0301 version).
Training
The training configurations are provided in the table below.
The training takes place on 8x A100(80G) GPUs and lasts for around 4 hours at a cost of $48 using Lambda Labs.
We used DeepSpeed with fully sharded data parallelism, also known as ZeRO stage 3 by writing our own fine-tuning scripts and leveraging some of the model training code provided by the amazing OpenAlpaca repo.
Here are some of the params used during training:
Property |
Details |
batch_size |
64 |
train_micro_batch_size_per_gpu |
4 |
gradient_accumulation_steps |
2 |
Learning rate |
2e-5 |
Max length |
1024 |
Epochs |
3 |
Optimizer |
AdamW |
Next Goals
- Try more data like actually using FLAN-v2, just like the Orca Research Paper (I am open for suggestions).
- Provide more options for Text generation UI (e.g., https://github.com/oobabooga/text-generation-webui).
- Provide 4bit GGML/GPTQ quantized model (perhaps TheBloke can help here).
Limitations & Biases
This model can produce factually incorrect output and should not be relied on to produce factually accurate information.
This model was trained on various public datasets. While great efforts have been taken to clean the pretraining data, it is possible that this model could generate lewd, biased, or otherwise offensive outputs.
Disclaimer
The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model.
Please consult an attorney before using this model for commercial purposes.
Citation
If you found wizardlm_alpaca_dolly_orca_open_llama_3b useful in your research or applications, please kindly cite using the following BibTeX:
@misc{orca_mini_3b,
author = {Pankaj Mathur},
title = {wizardlm_alpaca_dolly_orca_open_llama_3b: An explain tuned OpenLLaMA-3b model on custom wizardlm, alpaca, & dolly datasets},
year = {2023},
publisher = {GitHub, HuggingFace},
journal = {GitHub repository, HuggingFace repository},
howpublished = {\url{https://github.com/pankajarm/wizardlm_alpaca_dolly_orca_open_llama_3b}, \url{https://https://huggingface.co/psmathur/wizardlm_alpaca_dolly_orca_open_llama_3b}},
}
@misc{mukherjee2023orca,
title={Orca: Progressive Learning from Complex Explanation Traces of GPT-4},
author={Subhabrata Mukherjee and Arindam Mitra and Ganesh Jawahar and Sahaj Agarwal and Hamid Palangi and Ahmed Awadallah},
year={2023},
eprint={2306.02707},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
@software{openlm2023openllama,
author = {Xinyang Geng and Hao Liu},
title = {OpenLLaMA: An Open Reproduction of LLaMA},
month = May,
year = 2023,
url = {https://github.com/openlm-research/open_llama}
}
@misc{openalpaca,
author = {Yixuan Su and Tian Lan and Deng Cai},
title = {OpenAlpaca: A Fully Open-Source Instruction-Following Model Based On OpenLLaMA},
year = {2023},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/yxuansu/OpenAlpaca}},
}
@misc{alpaca,
author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
title = {Stanford Alpaca: An Instruction-following LLaMA model},
year = {2023},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
}
Detailed results can be found here
Property |
Details |
Avg. |
35.5 |
ARC (25-shot) |
41.55 |
HellaSwag (10-shot) |
61.52 |
MMLU (5-shot) |
26.79 |
TruthfulQA (0-shot) |
42.42 |
Winogrande (5-shot) |
61.8 |
GSM8K (5-shot) |
0.08 |
DROP (3-shot) |
14.33 |
Detailed results can be found here
Property |
Details |
Avg. |
39.03 |
AI2 Reasoning Challenge (25-Shot) |
41.55 |
HellaSwag (10-Shot) |
61.52 |
MMLU (5-Shot) |
26.79 |
TruthfulQA (0-shot) |
42.42 |
Winogrande (5-shot) |
61.80 |
GSM8k (5-shot) |
0.08 |
đ License
This model is licensed under cc-by-nc-sa-4.0.