Pixart

🚀 pixart-controlnet-lora-test

This project is a ControlNet PEFT LoRA derived from the base model, which aims to generate high - quality photo - realistic images, such as cat images.

🚀 Quick Start

This is a ControlNet PEFT LoRA derived from terminusresearch/pixart-900m-1024-ft-v0.6.

The main validation prompt used during training was:

A photo-realistic image of a cat

✨ Features

• The text encoder was not trained. You may reuse the base model text encoder for inference.
• You can find some example images in the following gallery:

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

import torch
from diffusers import PixArtSigmaPipeline, PixArtSigmaControlNetPipeline
# if you're not in the SimpleTuner environment, this import will fail.
from helpers.models.pixart.controlnet import PixArtSigmaControlNetAdapterModel

# Load base model
base_model_id = "terminusresearch/pixart-900m-1024-ft-v0.6"
controlnet_id = "bghira/pixart-controlnet-lora-test"

# Load ControlNet adapter
controlnet = PixArtSigmaControlNetAdapterModel.from_pretrained(
    f"{controlnet_id}/controlnet"
)

# Create pipeline
pipeline = PixArtSigmaControlNetPipeline.from_pretrained(
    base_model_id,
    controlnet=controlnet,
    torch_dtype=torch.bfloat16
)
pipeline.to('cuda' if torch.cuda.is_available() else 'cpu')

# Load your control image
from PIL import Image
control_image = Image.open("path/to/control/image.png")

# Generate
prompt = "A photo-realistic image of a cat"
image = pipeline(
    prompt=prompt,
    image=control_image,
    num_inference_steps=16,
    guidance_scale=4.0,
    generator=torch.Generator(device='cuda').manual_seed(42),
    controlnet_conditioning_scale=1.0,
).images[0]

image.save("output.png")

📚 Documentation

Validation settings

Note: The validation settings are not necessarily the same as the training settings.

Training settings

Datasets - antelope-data-1024

🔧 Technical Details

The project is based on the terminusresearch/pixart-900m-1024-ft-v0.6 base model, using ControlNet PEFT LoRA technology for training. Different settings are used in the validation and training processes, and specific parameters are provided in the corresponding sections. The text encoder is not trained, and the base model text encoder can be reused for inference.

📄 License

The license of this project is openrail++.