đ đŦ Einstein-v4-7B
This model is a fully fine-tuned variant of mistralai/Mistral-7B-v0.1 on a diverse set of datasets. It's finetuned using 7xRTX3090 + 1xRTXA6000 with axolotl, and the training was sponsored by sablo.ai.
⨠Features
- Full Fine-tuning: Based on mistralai/Mistral-7B-v0.1, fully fine-tuned on diverse datasets.
- Hardware Utilization: Utilizes
7xRTX3090 + 1xRTXA6000 for finetuning.
- Sponsorship: Training sponsored by sablo.ai.
- Quantization: Quantized versions (GGUF, AWQ, Exl2) are available.
đĻ Installation
No installation steps are provided in the original document.
đģ Usage Examples
Basic Usage
You can use the following prompt template while using the model:
ChatML
<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{asistant}<|im_end|>
This prompt template is available as a chat template, which means you can format messages using the tokenizer.apply_chat_template() method:
messages = [
{"role": "system", "content": "You are helpful AI asistant."},
{"role": "user", "content": "Hello!"}
]
gen_input = tokenizer.apply_chat_template(message, return_tensors="pt")
model.generate(**gen_input)
đ Documentation
Model Information
| Property |
Details |
| Model Type |
Full fine-tuned version of mistralai/Mistral-7B-v0.1 |
| Training Data |
Diverse datasets including allenai/ai2_arc, camel-ai/physics, camel-ai/chemistry, etc. |
Axolotl Config
See axolotl config
axolotl version: 0.4.0
base_model: mistralai/Mistral-7B-v0.1
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
is_mistral_derived_model: true
load_in_8bit: false
load_in_4bit: false
strict: false
chat_template: chatml
datasets:
- path: data/merged_all.json
ds_type: json
type: alpaca
conversation: chatml
- path: data/capybara_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/synthia-v1.3_sharegpt_12500.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/cot_alpaca_gpt4_extracted_openhermes_2.5_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/slimorca_dedup_filtered_95k_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/airoboros_3.2_without_contextual_slimorca_orca_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
dataset_prepared_path: last_run_prepared
val_set_size: 0.005
output_dir: ./Einstein-v4-model
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
eval_sample_packing: false
wandb_project: Einstein
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
hub_model_id: Weyaxi/Einstein-v4-7B
save_safetensors: true
gradient_accumulation_steps: 4
micro_batch_size: 1
num_epochs: 1.5
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.000005
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 10
evals_per_epoch: 2
eval_table_size:
eval_table_max_new_tokens: 128
saves_per_epoch: 4
debug:
deepspeed: zero3_bf16.json
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
bos_token: "<s>"
eos_token: "<|im_end|>"
unk_token: "<unk>"
tokens:
- "<|im_start|>"
resume_from_checkpoint: Einstein-v4-model/checkpoint-521
Quantizationed Versions
Evaluation Results
Detailed results can be found here
| Metric |
Value |
| Avg. |
66.62 |
| AI2 Reasoning Challenge (25-Shot) |
64.68 |
| HellaSwag (10-Shot) |
83.75 |
| MMLU (5-Shot) |
62.31 |
| TruthfulQA (0-shot) |
55.15 |
| Winogrande (5-shot) |
76.24 |
| GSM8k (5-shot) |
57.62 |
Detailed results can be found here
| Metric |
Value |
| Avg. |
16.73 |
| IFEval (0-Shot) |
47.08 |
| BBH (3-Shot) |
14.30 |
| MATH Lvl 5 (4-Shot) |
1.74 |
| GPQA (0-shot) |
4.25 |
| MuSR (0-shot) |
19.02 |
| MMLU-PRO (5-shot) |
13.99 |
Resources, Discussions and Reviews
đĻ Announcement tweet:
https://twitter.com/Weyaxi/status/1765851433448944125
đ Reddit post in r/LocalLLaMA:
https://www.reddit.com/r/LocalLLaMA/comments/1b9gmvl/meet_einsteinv47b_mistralbased_sft_model_using/
âļī¸ Youtube Videos
đ§ Technical Details
This model is full fine-tuned for 1.5 epoch. The total number of steps was 1562.
Loss graph
đ License
The license is other.
đ¤ Acknowledgments
Thanks to sablo.ai for sponsoring this model.
Thanks to all the dataset authors mentioned in the datasets section.
Thanks to axolotl for making the repository I used to make this model.
Thanks to all open source AI community.
If you would like to support me:
â Buy Me a Coffee
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
