🚀 14B-Qwen2.5-Freya-v1
本项目围绕Qwen 2.5的基础模型进行微调训练，探索了多步训练等方法，生成了14B-Qwen2.5-Freya-v1模型。该模型在特定的文本数据上进行训练，为相关领域的应用提供了新的选择。
训练失败时的我
✨ 主要特性
Freya-S1
- 在Qwen 2.5基础模型上，基于约1.1GB的文学和原始文本进行LoRA训练。
- 尽可能对文本和文献进行了清理，但可能仍存在一些问题。
Freya-S2
- 先将第一个LoRA应用于Qwen 2.5 Instruct，然后在此基础上继续训练。
- 降低了LoRA的秩，主要是因为模型以指令学习为主，还有一些细节暂不赘述。
📦 安装指南
文档中未提及安装相关内容，暂无法提供安装指南。
💻 使用示例
推荐的模型设置
注意，以下是我使用的设置，效果还不错。我甚至不清楚DRY或其他采样器的工作原理。无论如何，你的系统提示更为重要。
Prompt Format: ChatML
Temperature: 1+ # 我也不太清楚，伙计。
min_p: 0.05
📚 详细文档
训练信息
- 训练总时长约为10小时，在8xH100节点上完成，由新加坡政府或其他机构赞助。感谢内政部的国民服役津贴。
- 如需联系，请访问：https://sao10k.carrd.co/
模型相关信息
| 属性 |
详情 |
| 模型名称 |
14B-Qwen2.5-Freya-v1 |
| 基础模型 |
Qwen/Qwen2.5-14B |
| 标签 |
generated_from_trainer |
| 许可证 |
qwen |
基于Axolotl构建
查看axolotl配置
axolotl版本: 0.6.0
base_model:
- s1: Qwen/Qwen2.5-14B
- s2: Qwen/Qwen2.5-14B-Instruct
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: false
strict: false
sequence_len: 16384
bf16: auto
fp16:
tf32: false
flash_attention: true
special_tokens:
adapter: lora
lora_r:
- s1: 64
- s2: 32
lora_alpha: 64
lora_dropout: 0.2
lora_fan_in_fan_out:
peft_use_rslora: true
lora_target_linear: true
dataset_prepared_path: dataset_run_freya
datasets:
- path: datasets/eBooks-cleaned-75K
type: completion
- path: datasets/novels-clean-dedupe-10K
type: completion
- path: datasets/10k-amoral-full-fixed-sys.json
type: chat_template
chat_template: chatml
roles_to_train: ["gpt"]
field_messages: conversations
message_field_role: from
message_field_content: value
train_on_eos: turn
- path: datasets/44k-hespera-smartshuffle.json
type: chat_template
chat_template: chatml
roles_to_train: ["gpt"]
field_messages: conversations
message_field_role: from
message_field_content: value
train_on_eos: turn
- path: datasets/5k_rpg_adventure_instruct-sys.json
type: chat_template
chat_template: chatml
roles_to_train: ["gpt"]
field_messages: conversations
message_field_role: from
message_field_content: value
train_on_eos: turn
shuffle_merged_datasets: true
warmup_ratio: 0.1
plugins:
- axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_layer_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true
num_epochs:
- s1: 1
- s2: 2
sample_packing: true
pad_to_sequence_len: true
train_on_inputs: false
group_by_length: false
gradient_accumulation_steps: 4
micro_batch_size: 2
gradient_checkpointing: unsloth
val_set_size: 0.025
evals_per_epoch: 5
eval_table_size:
eval_max_new_tokens: 256
eval_sample_packing: false
eval_batch_size: 1
optimizer: paged_ademamix_8bit
lr_scheduler: cosine
learning_rate:
- s1: 0.000002
- s2: 0.000004
weight_decay: 0.2
max_grad_norm: 10.0
gc_steps: 10
deepspeed: ./deepspeed_configs/zero2.json
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
Transformers 支持多种语言