Run Stable Diffusion XL on RTX 3090 (Complete Setup)

VRAM Check: SDXL on RTX 3090

Stable Diffusion XL (SDXL) is the industry-standard open image generation model. The RTX 3090 with 24 GB VRAM is an excellent card for running it on a dedicated GPU server, especially for Stable Diffusion hosting.

The RTX 3090 handles every SDXL configuration comfortably, including full base + refiner pipelines with ControlNet and LoRA adapters. You even have enough headroom for 2048×2048 upscaled generations.

Setup with Diffusers

# Install dependencies
pip install diffusers transformers accelerate torch

# Generate an image with SDXL
from diffusers import StableDiffusionXLPipeline
import torch

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16"
).to("cuda")

image = pipe(
    prompt="A futuristic GPU server room, cyberpunk style, neon lighting",
    num_inference_steps=30,
    guidance_scale=7.5
).images[0]
image.save("output.png")

For a production API, wrap the pipeline in FastAPI or use the Diffusers API server. Compare this with the newer Flux model in our Run Flux.1 on RTX 3090 guide.

Setup with ComfyUI

# Clone ComfyUI
git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI
pip install -r requirements.txt

# Download SDXL base model
wget -P models/checkpoints/ \
  https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors

# Launch ComfyUI
python main.py --listen 0.0.0.0 --port 8188

ComfyUI provides a node-based workflow editor accessible via web browser, ideal for iterating on complex generation pipelines with ControlNet, IP-Adapter, and custom LoRAs.

Generation Speed Benchmarks

Tested on RTX 3090, 1024×1024, 30 steps, Euler scheduler. See our benchmarks hub for more GPU comparisons.

At 14 images per minute for base generation, the RTX 3090 is fast enough for real-time production workloads. Even with the refiner pipeline, you get nearly 8 images per minute.

Optimisation Tips

• Use torch.compile() with PyTorch 2.x for a 20-30% speedup on repeated generations.
• Enable xformers or use PyTorch’s native SDPA for memory-efficient attention.
• VAE tiling reduces VRAM at higher resolutions. Enable with pipe.vae.enable_tiling().
• Sequential CPU offloading frees VRAM for other models if you are co-hosting with an LLM.
• Batch generation: generate 2-4 images at once to maximise GPU utilisation.

For cost estimation, use our cost calculator. Read the self-host guide for general server setup.

Next Steps

Consider pairing SDXL with an LLM for prompt enhancement, both fit on a single RTX 3090. For the next generation of image models, see our Flux.1 hosting page. Compare SDXL performance on different GPUs in our GPU comparisons section. For a budget alternative, check the RTX 4060 for lighter SD models.