Wizardlaker 7B

🚀 WizardLaker 7B

This is a merged model that combines the new WizardLM 2 7B model with a custom DolphinLake Model, showing promising performance.

This is a merge of the new WizardLM 2 7B model with my custom DolphinLake Model(https://huggingface.co/Noodlz/DolphinLake-7B). It seems to perform well and I'll be submitting it for evaluations on the openLLM leaderboards. This model was created using mergekit.

📚 Documentation

✨ Merge Details

🔧 Merge Method

This model was merged using the DARE TIES merge method, with amazingvince/Not-WizardLM-2-7B as the base.

📦 Models Merged

The following models were included in the merge:

• /Noodlz/DolphinLake-7B

📋 Configuration

The following YAML configuration was used to produce this model:

merge_method: dare_ties

parameters:
  int8_mask: true
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5 # fallback for rest of tensors
  embed_slerp: true

models:
  - model: amazingvince/Not-WizardLM-2-7B
    # No parameters necessary for base model
  - model: /Noodlz/DolphinLake-7B
    parameters:
      density: 0.58
      weight: 0.4

base_model: amazingvince/Not-WizardLM-2-7B

tokenizer_source: model:amazingvince/Not-WizardLM-2-7B
dtype: bfloat16

📄 License

This project is licensed under the apache-2.0 license.