đ tiny-ko-124m-sft-muon
This model is a fine - tuned version of minpeter/tiny-ko-124m-base-muon. It is trained on multiple datasets including HuggingFaceTB/smol-smoltalk, trillionlabs/multisystem-curated, allenai/tulu-3-sft-personas-instruction-following, lemon-mint/smol-koreantalk, lemon-mint/Korean-FineTome-100k, heegyu/open-korean-instructions-v20231020, coastral/korean-writing-style-instruct, and devngho/korean-instruction-mix. It achieves a loss of 1.6461 on the evaluation set.
See axolotl config
axolotl version: 0.12.0.dev0
base_model: minpeter/tiny-ko-124m-base-muon
hub_model_id: minpeter/tiny-ko-124m-sft-muon
output_dir: ./outputs/tiny-ko-124m-sft-muon
wandb_project: "axolotl"
wandb_entity: "kasfiekfs-e"
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
strict: false
chat_template: chatml
datasets:
- path: HuggingFaceTB/smol-smoltalk
type: chat_template
split: train
field_messages: messages
message_property_mappings:
role: role
content: content
- path: trillionlabs/multisystem-curated
type: chat_template
split: train
field_messages: messages
message_property_mappings:
role: role
content: content
- path: allenai/tulu-3-sft-personas-instruction-following
type: chat_template
split: train
field_messages: messages
message_property_mappings:
role: role
content: content
- path: lemon-mint/smol-koreantalk
type: chat_template
split: train
field_messages: messages
message_property_mappings:
role: role
content: content
- path: lemon-mint/Korean-FineTome-100k
type: chat_template
split: train
field_messages: messages
message_property_mappings:
role: role
content: content
- path: heegyu/open-korean-instructions-v20231020
type: chat_template
split: train
field_messages: conversations
message_property_mappings:
role: from
content: value
roles:
user: ["human", "user"]
assistant: ["gpt", "assistant", "bot"]
system: ["system", "input"]
- path: coastral/korean-writing-style-instruct
type: chat_template
split: train
field_messages: conversations
message_property_mappings:
role: from
content: value
- path: devngho/korean-instruction-mix
type: chat_template
split: train
field_messages: messages
message_property_mappings:
role: from
content: value
dataset_prepared_path: last_run_prepared
val_set_size: 0.001
save_safetensors: true
sequence_len: 2048
sample_packing: false
pad_to_sequence_len: false
use_pose: true
pose_max_context_len: 65536
overrides_of_model_config:
rope_theta: 10000.0
max_position_embeddings: 65536
gradient_accumulation_steps: 8
micro_batch_size: 32
num_epochs: 1
optimizer: muon
lr_scheduler: cosine
learning_rate: 3e-4
train_on_inputs: false
group_by_length: false
bf16: true
fp16:
tf32: true
gradient_checkpointing: false
gradient_checkpointing_kwargs:
use_reentrant: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
sdp_attention:
s2_attention:
save_steps: 200
warmup_steps: 20
eval_steps: 200
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
đ Quick Start
This model is a fine - tuned version of minpeter/tiny-ko-124m-base-muon on multiple datasets. It achieves a loss of 1.6461 on the evaluation set.
đ Documentation
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
đ§ Technical Details
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- distributed_type: multi - GPU
- num_devices: 4
- gradient_accumulation_steps: 8
- total_train_batch_size: 1024
- total_eval_batch_size: 128
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e - 08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 20
- training_steps: 6865
Training results
| Training Loss |
Epoch |
Step |
Validation Loss |
| No log |
0 |
0 |
2.4581 |
| 1.8892 |
0.1165 |
200 |
1.9059 |
| 1.802 |
0.2331 |
400 |
1.8333 |
| 1.7906 |
0.3496 |
600 |
1.7918 |
| 1.7761 |
0.4661 |
800 |
1.7638 |
| 1.7145 |
0.5827 |
1000 |
1.7423 |
| 1.7114 |
0.6992 |
1200 |
1.7255 |
| 1.6798 |
0.8157 |
1400 |
1.7123 |
| 1.6722 |
0.9323 |
1600 |
1.7006 |
| 1.6821 |
1.0484 |
1800 |
1.6928 |
| 1.6414 |
1.1649 |
2000 |
1.6864 |
| 1.6473 |
1.2814 |
2200 |
1.6794 |
| 1.6202 |
1.3980 |
2400 |
1.6729 |
| 1.6141 |
1.5145 |
2600 |
1.6689 |
| 1.6415 |
1.6310 |
2800 |
1.6645 |
| 1.6165 |
1.7476 |
3000 |
1.6603 |
| 1.6292 |
1.8641 |
3200 |
1.6573 |
| 1.6277 |
1.9806 |
3400 |
1.6541 |
| 1.6033 |
2.0967 |
3600 |
1.6537 |
| 1.6432 |
2.2133 |
3800 |
1.6517 |
| 1.602 |
2.3298 |
4000 |
1.6505 |
| 1.6435 |
2.4463 |
4200 |
1.6493 |
| 1.5941 |
2.5629 |
4400 |
1.6481 |
| 1.594 |
2.6794 |
4600 |
1.6473 |
| 1.5986 |
2.7959 |
4800 |
1.6468 |
| 1.586 |
2.9125 |
5000 |
1.6464 |
| 1.6146 |
3.0286 |
5200 |
1.6462 |
| 1.5985 |
3.1451 |
5400 |
1.6462 |
| 1.574 |
3.2616 |
5600 |
1.6462 |
| 1.5823 |
3.3782 |
5800 |
1.6460 |
| 1.597 |
3.4947 |
6000 |
1.6460 |
| 1.5859 |
3.6112 |
6200 |
1.6460 |
| 1.5769 |
3.7277 |
6400 |
1.6459 |
| 1.572 |
3.8443 |
6600 |
1.6459 |
| 1.6111 |
3.9608 |
6800 |
1.6461 |
Framework versions
- Transformers 4.53.1
- Pytorch 2.6.0+cu124
- Datasets 3.6.0
- Tokenizers 0.21.1
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
