L3.3-70B-Euryale-v2.3 Open-source Large Language Model - Free Support for Conversations and Creative Content Creation

L3.3 70B Euryale V2.3

Developed by Sao10K

A 70B-parameter large language model trained based on Llama 3.3 Instruct, specializing in dialogue generation and creative content creation

Large Language Model

Transformers

#Long text generation #Creative writing #Role-playing

Downloads 1,274

Release Time : 12/7/2024

Model Overview

Euryale v2.3 is a large language model trained on Llama 3.3 Instruct, using full-parameter fine-tuning instead of the popular Lora extraction method. The model excels in dialogue generation and creative content creation, supporting long-context processing (16k tokens).

Model Features

Full-parameter fine-tuning

Uses full-parameter fine-tuning instead of the popular Lora extraction method, potentially achieving better model performance

Long-context support

Supports context lengths of up to 16,384 tokens, suitable for processing long documents and complex dialogues

Multi-dataset fusion training

Incorporates non-aligned data, role-playing data, and instruction data to enhance model diversity and adaptability

Optimized inference settings

Provides validated recommended inference parameters (temperature 1.1, min_p 0.1) to ensure generation quality

Model Capabilities

Text generation

Dialogue systems

Creative writing

Role-playing

Instruction following

Use Cases

Creative writing

Story creation

Generates coherent and creative storylines and character dialogues

Dialogue systems

Intelligent assistant

Functions as an intelligent dialogue assistant, providing information queries and daily communication features

Role-playing

Character interaction

Simulates specific character language styles and behavioral patterns for interaction

🚀 L3.3-70B-Euryale-v2.3

A direct replacement / successor to Euryale v2.2, offering enhanced performance.

eury

🚀 Quick Start

This model is a direct replacement or successor to Euryale v2.2. In the author's opinion, it performs slightly better than Euryale v2.2 and Hanami - x1. It is entirely trained on top of Llama 3.3 Instruct, rather than being Lora - extracted.

Recommended Model Settings

Look, I just use these, they work fine enough. I don't even know how DRY or other meme samplers work. Your system prompt matters more anyway.

Prompt Format: Llama-3-Instruct
Temperature: 1.1
min_p: 0.1

✨ Features

Direct Successor: A direct replacement for Euryale v2.2.
Trained on Llama 3.3 Instruct: Entirely trained on top of Llama 3.3 Instruct, not Lora - extracted.

🔧 Technical Details

Model Information

Property	Details
Library Name	transformers
License	llama3
Base Model	meta - llama/Llama - 3.3 - 70B - Instruct
Model Name	L3.3 - 70B - Euryale - v2.3
Tags	generated_from_trainer

Axolotl Configuration

See axolotl config

axolotl version: 0.5.2

base_model: meta-llama/Llama-3.3-70B-Instruct
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
load_in_4bit: false
strict: false
sequence_len: 16384
bf16: auto
fp16:
tf32: false
flash_attention: true

adapter: lora
lora_model_dir:
lora_r: 128
lora_alpha: 16
lora_dropout: 0.1
lora_target_linear: true
lora_fan_in_fan_out:
peft_use_rslora: true

# Data
dataset_prepared_path: last_run_prepared
datasets:
  - path: datasets/amoral-full-sys-prompt.json # Unalignment Data - Cleaned Up from Original, Split to its own file
    type: customllama3
  - path: datasets/mimi-superfix-RP-filtered-fixed.json # RP / Creative-Instruct Data
    type: customllama3
  - path: datasets/hespera-smartshuffle.json # Hesperus-v2-Instruct Data
    type: customllama3
warmup_steps: 15

plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_layer_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true

# Iterations
num_epochs: 1

# Batching
gradient_accumulation_steps: 4
micro_batch_size: 1
gradient_checkpointing: "unsloth"

# Optimizer
optimizer: paged_ademamix_8bit
lr_scheduler: cosine
learning_rate: 0.000004
weight_decay: 0.1
max_grad_norm: 25.0

# Iterations
num_epochs: 1

# Misc
deepspeed: ./deepspeed_configs/zero3_bf16.json

📄 License

This project is under the llama3 license.

Future Plans

Further refine the Datasets used for quality, including more secondary chats and more creative - related domains.
Work on other incomplete projects. There are about half a dozen projects on the backburner for a while now.

Acknowledgments

Special thanks to the author's wallet for funding this, juniors who share a single braincell between them, and the current national service.

Contact

For contact, visit https://sao10k.carrd.co/.

Art by てぃあ
https://www.pixiv.net/en/users/724263

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご