Flux Dev Openpose Controlnet

🚀 OpenPose ControlNet for Flux.dev

This is an OpenPose ControlNet for Flux.dev, trained on a specific dataset to enhance image generation capabilities.

🚀 Quick Start

✨ Features

• An OpenPose ControlNet trained on the dataset from raulc0399/open_pose_controlnet.
• Compatible with the xlabs ai pipeline from XLabs-AI/x-flux.

📦 Installation

To install the pipeline, execute the following commands:

git clone https://github.com/XLabs-AI/x-flux.git
cd x-flux
python3 -m venv xflux_env
source xflux_env/bin/activate
pip install -r requirements.txt

💻 Usage Examples

Basic Usage

To run the pipeline with ControlNet:

python3 main.py \
 --prompt "person enjoying a day at the park, full hd, cinematic" \
 --image ~/open_pose_controlnet_dataset/validation_images/pose/3_pose_1024.jpg --control_type openpose \
 --local_path ./model.safetensors \
 --use_controlnet --model_type flux-dev \
 --width 1024 --height 1024  --timestep_to_start_cfg 2 \
 --num_steps 50 --true_gs 4 --guidance 4 \
 --save_path ~/gen_imgs

If the image has already been preprocessed, comment out the line #146 from src/flux/xflux_pipeline.py:

# self.annotator = Annotator(control_type, self.other_device)

Training

oxen clone https://hub.oxen.ai/raulc/open_pose_controlnet_dataset
git clone https://github.com/raulc0399/x-flux.git
cd x-flux
git checkout open_pose_training
python3 -m venv xflux_env
source xflux_env/bin/activate
pip install -r requirements.txt
huggingface-cli login
accelerate config
mkdir images
rsync -r ~/open_pose_controlnet_dataset/train/images/ images/
cp train_configs/test_openpose_controlnet.yaml train_configs/openpose_controlnet.yaml
accelerate launch train_flux_deepspeed_controlnet.py --config "train_configs/openpose_controlnet.yaml"

⚠️ Important Note

• Check the file train_configs/openpose_controlnet.yaml before starting.
• rsync is needed, cp does not work with that many files.
• The oxen repo has the caption files as json as expected by the training script.

Results

Using these 2 control images:

With these prompts:

• "two friends sitting by each other enjoying a day at the park, full hd, cinematic"
• "person enjoying a day at the park, full hd, cinematic"

Resulted in these generated images:

📄 License

Weights fall under the FLUX.1 [dev] Non-Commercial License.

Information Table