The Open - source Model of t2iadapter_sketch_sd15v2: Providing Sketch - Conditioned Control for Stable Diffusion 1.5

T2iadapter Sketch Sd15v2

Developed by TencentARC

T2I Adapter is a network architecture that provides additional conditional control for Stable Diffusion models. This checkpoint is specifically designed to offer sketch-based conditional control for Stable Diffusion v1.5.

Image Generation OtherOpen Source License:Apache-2.0 #Sketch-controlled generation #Stable Diffusion adaptation #Line art to image

Downloads 2,243

Release Time : 7/14/2023

Model Overview

A T2I Adapter trained on PidiNet edge detection, capable of generating images that match text descriptions based on hand-drawn black-and-white line art.

Model Features

Sketch-based conditional control

Precisely control the composition and outlines of generated images through hand-drawn line art.

Lightweight adapter

Enhances the generative control capabilities of the base model with minimal additional parameters.

Multimodal fusion

Simultaneously processes both text prompts and visual sketch inputs.

Model Capabilities

Sketch-to-image generation

Image style transfer

Precise composition control

Creative design assistance

Use Cases

Digital art creation

Concept design

Quickly generate high-quality concept art from rough sketches

Accelerates the design process while offering diverse style options

Illustration creation

Transform hand-drawn line art into complete artworks

Enhances visual effects while preserving original creative intent

Product design

Product prototype visualization

Convert design sketches into realistic renderings

Rapidly validates design concepts and reduces prototyping costs

🚀 T2I Adapter - Sketch

T2I Adapter is a network that provides additional conditioning to Stable Diffusion. Each T2I checkpoint takes a different type of conditioning as input and is used with a specific base Stable Diffusion checkpoint. This checkpoint provides conditioning on sketches for the Stable Diffusion 1.5 checkpoint.

🚀 Quick Start

Dependencies

pip install diffusers transformers controlnet_aux

Run code

import torch
from PIL import Image
from controlnet_aux import PidiNetDetector

from diffusers import (
    T2IAdapter,
    StableDiffusionAdapterPipeline
)

image = Image.open('./images/sketch_in.png')

processor = PidiNetDetector.from_pretrained('lllyasviel/Annotators')

sketch_image = processor(image)

sketch_image.save('./images/sketch.png')

adapter = T2IAdapter.from_pretrained("TencentARC/t2iadapter_zoedepth_sd15v1", torch_dtype=torch.float16)
pipe = StableDiffusionAdapterPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", adapter=adapter, safety_checker=None, torch_dtype=torch.float16, variant="fp16"
)

pipe.to('cuda')

generator = torch.Generator().manual_seed(0)

sketch_image_out = pipe(prompt="royal chamber with fancy bed", image=sketch_image, generator=generator).images[0]

sketch_image_out.save('./images/sketch_image_out.png')

sketch_in sketch sketch_image_out

✨ Features

T2I Adapter provides additional conditioning to Stable Diffusion, allowing for more controllable text-to-image generation. This specific checkpoint enables conditioning on sketches for the Stable Diffusion 1.5 model.

📚 Documentation

Model Details

Property	Details
Developed by	T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
Model Type	Diffusion-based text-to-image generation model
Language(s)	English
License	Apache 2.0
Resources for more information	GitHub Repository, Paper.
Cite as	@misc{ title={T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models}, author={Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, Xiaohu Qie}, year={2023}, eprint={2302.08453}, archivePrefix={arXiv}, primaryClass={cs.CV} }

Checkpoints

Model Name	Control Image Overview	Control Image Example	Generated Image Example
TencentARC/t2iadapter_color_sd14v1 Trained with spatial color palette	A image with 8x8 color palette.
TencentARC/t2iadapter_canny_sd14v1 Trained with canny edge detection	A monochrome image with white edges on a black background.
TencentARC/t2iadapter_sketch_sd14v1 Trained with PidiNet edge detection	A hand-drawn monochrome image with white outlines on a black background.
TencentARC/t2iadapter_depth_sd14v1 Trained with Midas depth estimation	A grayscale image with black representing deep areas and white representing shallow areas.
TencentARC/t2iadapter_openpose_sd14v1 Trained with OpenPose bone image	A OpenPose bone image.
TencentARC/t2iadapter_keypose_sd14v1 Trained with mmpose skeleton image	A mmpose skeleton image.
TencentARC/t2iadapter_seg_sd14v1 Trained with semantic segmentation	An custom segmentation protocol image.
TencentARC/t2iadapter_canny_sd15v2
TencentARC/t2iadapter_depth_sd15v2
TencentARC/t2iadapter_sketch_sd15v2
TencentARC/t2iadapter_zoedepth_sd15v1

📄 License

This model is licensed under the Apache 2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご