ControlNet Union Self-Hosted

Running multiple ControlNets historically meant loading multiple full ControlNet weights. ControlNet Union (Xinsir’s unified model) handles canny, depth, openpose, scribble, soft edge, segmentation, normal, and lineart from one set of weights. On our dedicated GPU hosting it slashes the VRAM cost of multi-condition pipelines.

Contents

• VRAM savings
• Deployment
• Mode selection
• Quality vs specialised

VRAM

Traditional SDXL ControlNet: ~2.5 GB per mode. Three modes = 7.5 GB. ControlNet Union: one model handling all modes at ~2.5 GB total. Meaningful saving when you need multiple conditioners.

Deployment

from diffusers import StableDiffusionXLControlNetUnionPipeline, ControlNetUnionModel
import torch

controlnet = ControlNetUnionModel.from_pretrained(
    "xinsir/controlnet-union-sdxl-1.0",
    torch_dtype=torch.float16,
)

pipe = StableDiffusionXLControlNetUnionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet,
    torch_dtype=torch.float16,
).to("cuda")

image = pipe(
    prompt="a detailed portrait",
    control_image=[canny_image, depth_image],
    control_mode=[0, 1],  # 0=canny, 1=depth
    controlnet_conditioning_scale=[0.8, 0.5],
).images[0]

Modes

Quality

ControlNet Union quality is typically within 5-10% of dedicated single-purpose ControlNets. For most production workloads the VRAM saving outweighs the small quality gap. For workflows demanding the absolute best depth or pose control, specialised ControlNets still edge Union out.

See IP-Adapter and InstantID.