đ đŦ Einstein-v7-Qwen2-7B
This model is a fully fine-tuned version of Qwen/Qwen2-7B on diverse datasets. It's finetuned using 8xMI300X with axolotl and trained with compute resources from TensorWave.
đ Quick Start
Model Usage
You can use the ChatML prompt template when using the model. The template is available as a chat template, allowing you to format messages with the tokenizer.apply_chat_template() method.
ChatML Prompt Template
<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{asistant}<|im_end|>
Python Code Example
messages = [
{"role": "system", "content": "You are helpful AI asistant."},
{"role": "user", "content": "Hello!"}
]
gen_input = tokenizer.apply_chat_template(message, return_tensors="pt")
model.generate(**gen_input)
⨠Features
- Full Fine - Tuning: Based on the Qwen/Qwen2-7B base model, it has been fully fine - tuned on a wide range of datasets.
- Diverse Datasets: Trained on various datasets related to science, math, and general instruction, ensuring comprehensive knowledge and strong generalization ability.
- ChatML Support: Supports the ChatML prompt template, making it easy to interact with the model.
đĻ Installation
No specific installation steps are provided in the original document.
đģ Usage Examples
Basic Usage
messages = [
{"role": "system", "content": "You are helpful AI asistant."},
{"role": "user", "content": "Hello!"}
]
gen_input = tokenizer.apply_chat_template(message, return_tensors="pt")
model.generate(**gen_input)
đ Documentation
Datasets Used
The datasets used to train this model are listed in the metadata section of the model card. Note that some datasets in the metadata may have been filtered according to different criteria. The filtering results are in a different repository: Weyaxi/sci - datasets/main
Quantizationed Versions
- https://huggingface.co/bartowski/Einstein-v7-Qwen2-7B-GGUF
- https://huggingface.co/bartowski/Einstein-v7-Qwen2-7B-exl2
Evaluation Results
The model's evaluation results on the Open LLM Leaderboard v2 are as follows:
| Metric |
Value |
| Avg. |
24.01 |
| IFEval (0 - Shot) |
41.00 |
| BBH (3 - Shot) |
32.84 |
| MATH Lvl 5 (4 - Shot) |
15.18 |
| GPQA (0 - shot) |
6.60 |
| MuSR (0 - shot) |
14.06 |
| MMLU - PRO (5 - shot) |
34.40 |
Detailed results can be found here
Additional Resources
Training Information
This model is fully fine - tuned for 2 epochs, with a total of 500 steps.
Loss graph
đ§ Technical Details
Axolotl Config
See axolotl config
axolotl version: 0.4.0
base_model: Qwen/Qwen2-7B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: false
strict: false
chat_template: chatml
datasets:
- path: data/airoboros_3.2_without_contextual_slimorca_orca_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/allenai_wild_chat_gpt4_english_toxic_random_half_4k_sharegpt.json
ds_type: json
type: sharegpt
strict: false
conversation: chatml
- path: data/buzz_unstacked_chosen_math_removed_filtered.json
ds_type: json
type: alpaca
conversation: chatml
- path: data/capybara_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/cot_alpaca_gpt4_extracted_openhermes_2.5_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/everythinglm-data-v3_sharegpt.json
ds_type: json
type: sharegpt
strict: false
conversation: chatml
- path: data/gpt4_data_lmys_1m_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/gpteacher-instruct-special-alpaca.json
ds_type: json
type: gpteacher
conversation: chatml
- path: data/merged_all.json
ds_type: json
type: alpaca
conversation: chatml
- path: data/no_robots_sharegpt.json
ds_type: json
type: sharegpt
strict: false
conversation: chatml
- path: data/oasst_top1_from_fusechatmixture_sharegpt.json
ds_type: json
type: sharegpt
strict: false
conversation: chatml
- path: data/pippa_bagel_repo_3k_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/rpguild_quarter_alignment_lab_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/sharegpt_gpt4_english.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/slimorca_dedup_filtered_95k_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/soda_diaolog_longest_tenth_buzz_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/synthia-v1.3_sharegpt_12500.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/system_conversations_dolphin_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
dataset_prepared_path: last_run_prepared
val_set_size: 0.002
output_dir: ./Einstein-v7-Qwen2-7B-model
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
eval_sample_packing: false
wandb_project: Einstein
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
hub_model_id: Weyaxi/Einstein-v7-Qwen2-7B
gradient_accumulation_steps: 4
micro_batch_size: 6
num_epochs: 2
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 0.00001
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: unsloth
gradient_checkpointing_kwargs:
use_reentrant: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 10
evals_per_epoch: 2
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed: deepspeed_configs/zero3_bf16.json
weight_decay: 0.05
fsdp:
fsdp_config:
special_tokens:
eos_token: "<|im_end|>"
pad_token: "<|end_of_text|>"
tokens:
- "<|im_start|>"
- "<|im_end|>"
đ License
The license of this model is other.
đ¤ Acknowledgments
Thanks to all the dataset authors mentioned in the datasets section.
Thanks to axolotl for providing the repository used to create this model.
Thanks to all open - source AI communities.
If you would like to support me:
â Buy Me a Coffee
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
