Batch Image Generation: GPU Throughput Optimization

Your GPU Is Generating One Image at a Time

You need thousands of images per hour, but your GPU server produces them sequentially — one prompt, one image, repeat. The GPU utilisation reported by nvidia-smi hovers at 60-70 percent between generations, meaning you are leaving significant throughput on the table. Batch generation done correctly can triple or quadruple your images-per-hour without upgrading hardware.

Native Batch Generation in Diffusers

The simplest optimisation is generating multiple images per forward pass. This fills the GPU’s compute units more completely:

from diffusers import StableDiffusionXLPipeline
import torch

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16, use_safetensors=True
).to("cuda")

# Single image: ~7.2 seconds on RTX 5090
image = pipe("a mountain landscape", num_inference_steps=25).images[0]

# Batch of 4: ~14.8 seconds total = ~3.7 seconds per image
prompts = ["a mountain landscape"] * 4
images = pipe(prompts, num_inference_steps=25).images

# Batch of 8 (needs ~20 GB VRAM for SDXL):
prompts = ["a mountain landscape"] * 8
images = pipe(prompts, num_inference_steps=25).images

VRAM consumption scales linearly with batch size. Find the largest batch that fits in your GPU memory, then step back by one for safety margin.

Finding the Optimal Batch Size

Run a sweep to find the throughput sweet spot for your specific GPU and model:

import time, torch
from diffusers import StableDiffusionXLPipeline

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16).to("cuda")

for batch_size in [1, 2, 4, 8, 12, 16]:
    torch.cuda.empty_cache()
    prompts = ["benchmark prompt"] * batch_size
    try:
        start = time.time()
        pipe(prompts, num_inference_steps=25)
        elapsed = time.time() - start
        ips = batch_size / elapsed
        vram = torch.cuda.max_memory_allocated() / 1e9
        print(f"Batch {batch_size:2d}: {ips:.2f} img/s, {vram:.1f} GB peak")
        torch.cuda.reset_peak_memory_stats()
    except torch.cuda.OutOfMemoryError:
        print(f"Batch {batch_size:2d}: OOM")
        break

Typical results on an RTX 5090 with SDXL at 1024×1024: batch 1 gives 0.14 img/s, batch 4 gives 0.27 img/s, batch 8 OOMs. The sweet spot sits at batch 4.

Queue-Based Pipeline Architecture

For production systems receiving prompts via API, decouple request intake from generation:

import asyncio, torch
from collections import deque
from diffusers import StableDiffusionXLPipeline

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16).to("cuda")

BATCH_SIZE = 4
BATCH_TIMEOUT = 2.0  # seconds to wait for a full batch
request_queue = deque()

async def batch_worker():
    while True:
        # Collect up to BATCH_SIZE requests or wait BATCH_TIMEOUT
        batch = []
        deadline = asyncio.get_event_loop().time() + BATCH_TIMEOUT
        while len(batch) < BATCH_SIZE:
            remaining = deadline - asyncio.get_event_loop().time()
            if remaining <= 0:
                break
            try:
                item = await asyncio.wait_for(get_request(), remaining)
                batch.append(item)
            except asyncio.TimeoutError:
                break
        if batch:
            prompts = [r["prompt"] for r in batch]
            images = pipe(prompts, num_inference_steps=25).images
            for req, img in zip(batch, images):
                req["future"].set_result(img)

This pattern amortises pipeline overhead across multiple requests and keeps GPU utilisation consistently above 90 percent.

Multi-GPU Parallel Generation

With multiple GPUs, run independent pipeline instances rather than splitting a single generation:

# Launch one pipeline per GPU
import torch.multiprocessing as mp

def gpu_worker(gpu_id, prompt_queue, result_queue):
    pipe = StableDiffusionXLPipeline.from_pretrained(
        "stabilityai/stable-diffusion-xl-base-1.0",
        torch_dtype=torch.float16
    ).to(f"cuda:{gpu_id}")
    while True:
        batch = prompt_queue.get()
        if batch is None:
            break
        images = pipe(batch, num_inference_steps=25).images
        result_queue.put(images)

# 4 GPUs = 4x throughput with independent batching

Additional Throughput Gains

Layer these optimisations on top of batching for compounding improvements on your GPU server:

• torch.compile: Compiles the UNet for 15-30 percent speedup. Run pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead").
• xformers: Enables memory-efficient attention. pipe.enable_xformers_memory_efficient_attention().
• VAE tiling: For high-resolution output, pipe.vae.enable_tiling() prevents VAE decode OOM.
• Persistent workers: Keep the pipeline loaded between requests to eliminate reload overhead.

For ComfyUI batch workflows, use the batch node with queue mode. Stable Diffusion hosting at scale benefits from the queue architecture above. Check benchmarks for per-GPU throughput data, and our PyTorch guide for environment setup. The tutorials section covers model-specific tuning, and the PyTorch hosting page lists compatible hardware.