T2iadapter Zoedepth Sd15v1

🚀 T2I Adapter - Zoedepth

T2I Adapter是一個為穩定擴散模型提供額外條件的網絡。每個T2I檢查點接收不同類型的條件作為輸入，並與特定的基礎穩定擴散檢查點一起使用。

此檢查點為穩定擴散1.5檢查點提供基於ZoeDepth深度估計的條件。

🚀 快速開始

依賴安裝

pip install diffusers transformers matplotlib

運行代碼

from PIL import Image
import torch
import numpy as np
import matplotlib
from diffusers import T2IAdapter, StableDiffusionAdapterPipeline

def colorize(value, vmin=None, vmax=None, cmap='gray_r', invalid_val=-99, invalid_mask=None, background_color=(128, 128, 128, 255), gamma_corrected=False, value_transform=None):
    """Converts a depth map to a color image.

    Args:
        value (torch.Tensor, numpy.ndarry): Input depth map. Shape: (H, W) or (1, H, W) or (1, 1, H, W). All singular dimensions are squeezed
        vmin (float, optional): vmin-valued entries are mapped to start color of cmap. If None, value.min() is used. Defaults to None.
        vmax (float, optional):  vmax-valued entries are mapped to end color of cmap. If None, value.max() is used. Defaults to None.
        cmap (str, optional): matplotlib colormap to use. Defaults to 'magma_r'.
        invalid_val (int, optional): Specifies value of invalid pixels that should be colored as 'background_color'. Defaults to -99.
        invalid_mask (numpy.ndarray, optional): Boolean mask for invalid regions. Defaults to None.
        background_color (tuple[int], optional): 4-tuple RGB color to give to invalid pixels. Defaults to (128, 128, 128, 255).
        gamma_corrected (bool, optional): Apply gamma correction to colored image. Defaults to False.
        value_transform (Callable, optional): Apply transform function to valid pixels before coloring. Defaults to None.

    Returns:
        numpy.ndarray, dtype - uint8: Colored depth map. Shape: (H, W, 4)
    """
    if isinstance(value, torch.Tensor):
        value = value.detach().cpu().numpy()

    value = value.squeeze()
    if invalid_mask is None:
        invalid_mask = value == invalid_val
    mask = np.logical_not(invalid_mask)

    # normalize
    vmin = np.percentile(value[mask],2) if vmin is None else vmin
    vmax = np.percentile(value[mask],85) if vmax is None else vmax
    if vmin != vmax:
        value = (value - vmin) / (vmax - vmin)  # vmin..vmax
    else:
        # Avoid 0-division
        value = value * 0.

    # squeeze last dim if it exists
    # grey out the invalid values

    value[invalid_mask] = np.nan
    cmapper = matplotlib.cm.get_cmap(cmap)
    if value_transform:
        value = value_transform(value)
        # value = value / value.max()
    value = cmapper(value, bytes=True)  # (nxmx4)

    img = value[...]
    img[invalid_mask] = background_color

    if gamma_corrected:
        img = img / 255
        img = np.power(img, 2.2)
        img = img * 255
        img = img.astype(np.uint8)
    return img

model = torch.hub.load("isl-org/ZoeDepth", "ZoeD_N", pretrained=True)

img = Image.open('./images/zoedepth_in.png')

out = model.infer_pil(img)

zoedepth_image = Image.fromarray(colorize(out)).convert('RGB')

zoedepth_image.save('images/zoedepth.png')

adapter = T2IAdapter.from_pretrained("TencentARC/t2iadapter_zoedepth_sd15v1", torch_dtype=torch.float16)
pipe = StableDiffusionAdapterPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", adapter=adapter, safety_checker=None, torch_dtype=torch.float16, variant="fp16"
)

pipe.to('cuda')
zoedepth_image_out = pipe(prompt="motorcycle", image=zoedepth_image).images[0]

zoedepth_image_out.save('images/zoedepth_out.png')

✨ 主要特性

T2I Adapter網絡能夠為穩定擴散模型提供額外的條件輸入，每個T2I檢查點可接收不同類型的條件，與特定的基礎穩定擴散檢查點配合使用，增強了文本到圖像擴散模型的可控性。

📚 詳細文檔

模型詳情

• 開發者：T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
• 模型類型：基於擴散的文本到圖像生成模型
• 語言：英語
• 許可證：Apache 2.0
• 更多信息資源：GitHub倉庫，論文。
• 引用方式：

@misc{
    title={T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models}, 
    author={Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, Xiaohu Qie},
    year={2023},
    eprint={2302.08453},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

檢查點

模型名稱	控制圖像概述	控制圖像示例	生成圖像示例
TencentARC/t2iadapter_color_sd14v1 使用空間調色板訓練	一張8x8調色板的圖像。
TencentARC/t2iadapter_canny_sd14v1 使用Canny邊緣檢測訓練	一張黑色背景上帶有白色邊緣的單色圖像。
TencentARC/t2iadapter_sketch_sd14v1 使用PidiNet邊緣檢測訓練	一張黑色背景上帶有白色輪廓的手繪單色圖像。
TencentARC/t2iadapter_depth_sd14v1 使用Midas深度估計訓練	一張灰度圖像，黑色代表深區域，白色代表淺區域。
TencentARC/t2iadapter_openpose_sd14v1 使用OpenPose骨骼圖像訓練	一張OpenPose骨骼圖像。
TencentARC/t2iadapter_keypose_sd14v1 使用mmpose骨架圖像訓練	一張mmpose骨架圖像。
TencentARC/t2iadapter_seg_sd14v1 使用語義分割訓練	一張自定義分割協議圖像。
TencentARC/t2iadapter_canny_sd15v2
TencentARC/t2iadapter_depth_sd15v2
TencentARC/t2iadapter_sketch_sd15v2
TencentARC/t2iadapter_zoedepth_sd15v1

📄 許可證

本模型使用Apache 2.0許可證。