70B-L3.3-mhnnn-x1 Open-Source Large Language Model - Free Creative Text Generation and Multi-Task Processing

70B L3.3 Mhnnn X1

Developed by Sao10K

A large language model fine-tuned based on Llama-3-70B-Instruct, specializing in creative text generation and multi-task processing

Large Language Model

Transformers

#Creative Text Generation #Amoral Neutral Output #Multi-Scenario Adaptation

Downloads 150

Release Time : 1/1/2025

Model Overview

This model is fine-tuned from Meta's Llama-3-70B-Instruct, with special optimizations for creative text generation, suitable for various scenarios such as novel continuation, text adventure games, amoral assistants, and role-playing.

Model Features

High Creative Output

The model is specially optimized for creative text generation, suitable for novel writing and story continuation

Multi-Task Adaptation

Capable of handling various task types such as text adventure games, amoral assistants, and role-playing

Flexible Prompt Format

Supports Llama-3-Instruct prompt format, allowing model behavior adjustment through system prompts

Model Capabilities

Text Continuation

Dialogue Generation

Story Creation

Text Adventure Game Generation

Role-Playing Dialogue

Unrestricted Content Generation

Use Cases

Creative Writing

Novel Continuation

Generate coherent subsequent plotlines based on existing novel content

Produces creative storylines, occasionally requiring regeneration for optimal results

Game Development

Text Adventure Games

Generate narration and plot development for text adventure games

With just a single example demonstration, the model can generate coherent game content

Dialogue Systems

Amoral Assistant

Generate dialogue content without moral restrictions

Including terms like 'amoral' in prompts yields more unrestricted responses

🚀 70B-L3.3-mhnnn-x1

This model, 70B-L3.3-mhnnn-x1, is based on the meta-llama/Llama-3.3-70B-Instruct base model from the transformers library. After some experimentation, it shows promising performance. It shares the same data composition as Freya but applies it differently.

✨ Features

Creative Outputs: It can generate more creative outputs, although it may have occasional glitches that can be fixed by regeneration.
Multiple Data Types: It can handle various types of data, including novel/eBook completions, text adventures, amoral assistant tasks, instruct/assistant interactions, and role - playing.

📦 Installation

No installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

Prompt Format: Llama-3-Instruct
Temperature: 1.1
min_p: 0.05

Advanced Usage

Completion - Novels / eBooks
Text Adventure - Include details like 'Text Adventure Narrator' in the System Prompt, give it a one-shot example and it'll fly.
Amoral Assistant - Include the terms 'Amoral', 'Neutral' along with the regular assistant prompt for better results.
Instruct / Assistant - The usual assistant tasks.
Roleplay - As per Usual, Regular Sets

🔧 Technical Details

The model was trained for approximately 14 hours on an 8xH100 Node.

📄 License

The license for this model is llama3.3.

📚 Documentation

Property	Details
Library Name	transformers
Base Model	meta-llama/Llama-3.3-70B-Instruct
Model Name	70B-L3.3-mhnnn-x1
License	llama3.3

Other Information

yeah my mental when things do not go well

Training time in total was ~14 Hours on a 8xH100 Node, shout out to SCDF for not sponsoring this run. My funds are dry doing random things.

Contact me at https://sao10k.carrd.co/.

See axolotl config

axolotl version: 0.6.0

adapter: lora # 16-bit
lora_r: 64
lora_alpha: 64
lora_dropout: 0.2
peft_use_rslora: true
lora_target_linear: true
  
# Data
dataset_prepared_path: dataset_run_freya
datasets:
# S1 - Writing / Completion
  - path: datasets/eBooks-cleaned-75K
    type: completion
  - path: datasets/novels-clean-dedupe-10K
    type: completion
# S2 - Instruct
  - path: datasets/10k-amoral-full-fixed-sys.json
    type: chat_template
    chat_template: llama3
    roles_to_train: ["gpt"]
    field_messages: conversations
    message_field_role: from
    message_field_content: value
    train_on_eos: turn
  - path: datasets/44k-hespera-smartshuffle.json
    type: chat_template
    chat_template: llama3
    roles_to_train: ["gpt"]
    field_messages: conversations
    message_field_role: from
    message_field_content: value
    train_on_eos: turn
  - path: datasets/5k_rpg_adventure_instruct-sys.json
    type: chat_template
    chat_template: llama3
    roles_to_train: ["gpt"]
    field_messages: conversations
    message_field_role: from
    message_field_content: value
    train_on_eos: turn
shuffle_merged_datasets: true
warmup_ratio: 0.1

plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_layer_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true

# Iterations
num_epochs: 1

# Sampling
sample_packing: true
pad_to_sequence_len: true
train_on_inputs: false
group_by_length: false

# Batching
gradient_accumulation_steps: 4
micro_batch_size: 2
gradient_checkpointing: unsloth

# Evaluation
val_set_size: 0.025
evals_per_epoch: 5
eval_table_size:
eval_max_new_tokens: 256
eval_sample_packing: false
eval_batch_size: 1

# Optimizer
optimizer: paged_ademamix_8bit
lr_scheduler: cosine
learning_rate: 0.00000242
weight_decay: 0.2
max_grad_norm: 10.0

# Garbage Collection
gc_steps: 10

# Misc
deepspeed: ./deepspeed_configs/zero3_bf16.json

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご