RTX 4090 24GB SDXL Benchmark: 1024×1024 in 2.0s

SDXL 1.0 remains the most-deployed high-resolution diffusion model in production. The base UNet is 2.6 billion parameters in FP16, which is small by 2026 standards but pairs with two text encoders and produces native 1024-pixel output through a sufficiently parameterised pipeline that latency on lesser cards is painful. On an RTX 4090 24GB dedicated host from Gigagpu, a single 1024×1024 image at 30 steps with the DPM++ 2M Karras sampler renders in 2.0 seconds. A batch of four completes in 6.5 seconds, which is 1.63 seconds per image — the natural sweet spot for studio-grade serving with comfortable VRAM headroom for ControlNet and a stack of LoRAs.

Contents

• Methodology and test rig
• Single-image latency by resolution
• Batched throughput and VRAM
• VRAM map and the bandwidth ceiling
• Distilled variants: Lightning, Turbo, LCM
• Refiner, LoRA and ControlNet stacks
• Cross-card comparison
• Production gotchas
• Verdict: when to pick SDXL

Methodology and test rig

All measurements use Diffusers 0.30 with PyTorch 2.5, sdpa attention via F.scaled_dot_product_attention, on a stock RTX 4090 24GB Founders Edition at 450W TDP. Host is a Ryzen 9 7950X with 64GB DDR5-5600 and a Samsung 990 Pro 2TB Gen 4 NVMe; OS is Ubuntu 24.04 LTS, NVIDIA driver 560.x, CUDA 12.6. Each result is the median of fifty runs after a five-image warm-up; standard deviation under 3% across the run.

Wall-clock latency includes text encoding and VAE decode but excludes any network or PIL save time. Batch numbers reflect a single forward pass producing N images simultaneously, not N sequential calls. Where we use torch.compile, the 8-12% gain it produces is reported in the configuration section but excluded from the headline numbers, since first-call compilation latency makes it inappropriate for cold endpoints.

Single-image latency by resolution

SDXL’s quadratic attention scaling means resolution dominates latency once you cross the 1024 threshold. The numbers below cover the practical native-trained resolutions plus the common 1.5x landscape variants:

Note the cliff at 1536×1024 — the per-step time jumps from 67 ms to 103 ms because attention now operates on roughly 2.4x as many latent tokens. Above 1536 you should be running HiRes-fix or img2img upscaling rather than native generation.

Batched throughput and VRAM

Batching is the single highest-leverage optimisation for SDXL serving. The UNet weights stay resident; only activations and KV grow with batch. The 4090 has enough compute to push aggregate steps/s up to a practical ceiling around batch 4-6:

Batch 4 at 1024 is the production default. Batch 6 squeezes a fraction more throughput but leaves no headroom for ControlNet or text-encoder peaks. If you really need batch 8, drop to 896×896 (still above the 768 native trained resolution) or enable VAE tiling, which adds about 80 ms of latency but caps VAE peak around 2 GB.

VRAM map and the bandwidth ceiling

The 4090’s 24GB GDDR6X delivers 1008 GB/s of memory bandwidth and 16,384 CUDA cores at 2.5 GHz, which is enough that SDXL’s UNet is compute-bound, not memory-bound, at native resolutions. That changes the optimisation calculus relative to LLM workloads:

The math: each UNet step at 1024 touches roughly 5 GB of weights. At 1008 GB/s that is 5 ms of memory work, but the actual step takes 67 ms — the 4090 is doing real arithmetic, not just shuffling bytes. This is why FP8 quantisation of SDXL gives marginal wins (~10%) compared to the 40-50% wins it produces on bandwidth-bound LLM decode.

Distilled variants: Lightning, Turbo, LCM

For latency-critical paths, distilled SDXL variants reduce step counts dramatically. Quality drops are real but often acceptable for previews, thumbnails, or interactive sliders:

SDXL Turbo at 4 steps and 0.55 seconds gives you 109 images per minute single-stream, fast enough for interactive design tools. The full SDXL setup guide documents how to swap between base and distilled checkpoints behind a single endpoint for quality/speed tiering.

Refiner, LoRA and ControlNet stacks

Real production pipelines rarely run base-only. Adding refiner, LoRAs and ControlNet to the same 4090:

LoRAs are nearly free per-image because PEFT merges the low-rank deltas into the UNet weights at load time. The cost is one-off load latency (~150 ms per LoRA) and a small VRAM bump. ControlNet, by contrast, runs a parallel 1.3B network at every step — the +30% latency is unavoidable.

Cross-card comparison

SDXL 1024×1024 at 30 steps DPM++ 2M Karras, batch 1:

The 4090 sits in the price/perf sweet spot. The 5090 is faster but pricier per hour; the H100’s bandwidth advantage is wasted on a compute-bound workload. The 5060 Ti works for hobby use but multi-batch SDXL serving on it stops being economic above a few users.

Production gotchas

• VAE FP16 colour bug. The default SDXL VAE produces saturated artefacts in FP16 on certain images. Either use the madebyollin/sdxl-vae-fp16-fix checkpoint or run the VAE in FP32, which costs ~150 ms per decode.
• torch.compile cold start. First call after compile takes 90-120 seconds. Pre-warm before opening to traffic; persist the compiled cache between restarts via torch._inductor.config.fx_graph_cache = True.
• OpenCLIP-G memory spikes. The text encoder briefly allocates 3.4 GB during its forward pass. If you’re already at 22 GB UNet+activation, you’ll OOM on text encode. Encode prompts first, free encoders, then run UNet.
• ControlNet preprocessor on CPU is the bottleneck. Canny, depth and openpose preprocessing in PIL/OpenCV on CPU often takes 200-400 ms — longer than the actual generation. Move preprocessors to GPU (controlnet-aux supports this) or run them async.
• LoRA scale stacking is not commutative. Loading LoRA A at scale 0.8 then B at 0.6 produces different output than loading B then A. Always load in deterministic order and document the stack.
• Refiner needs the same noise schedule as base. Switching samplers between base and refiner produces seam artefacts at the handoff point. Stick to DPM++ 2M Karras for both.
• Don’t stream Diffusers callbacks over network. The intermediate latent decode adds 30-50 ms per callback; stream only every 5th step or you waste 20% of your latency budget on previews.

Verdict: when to pick SDXL on a 4090

SDXL on the 4090 is the workhorse choice for any production image pipeline that needs 1024-pixel native output, ControlNet support, and a mature LoRA ecosystem. At 1.63 seconds per image batched, you serve 2,200 images per hour from one card with comfortable headroom for stacking. Pick it over FLUX.1-dev when speed matters more than fidelity, over FLUX schnell when you need ControlNet (which the FLUX ecosystem is still catching up on), and over SD 1.5 whenever you’re outputting above 768 pixels. Move to the 5090 only when you’ve maxed out a 4090 and need batch 8 at 1024 native — and check best GPU for Stable Diffusion for the broader landscape first.

See also: Stable Diffusion setup, ComfyUI setup, FLUX schnell benchmark, FLUX dev benchmark, Stable Video Diffusion, image studio use case, 5060 Ti SDXL comparison, best GPU for SD.