RTX 4090 24GB Power Draw and Efficiency Under AI Inference

The RTX 4090 carries a 450 W TDP on the box, but on a steady-state inference server the time-averaged draw is rarely that high. Decode is bandwidth-bound and idles tensor cores; only batched prefill, image generation and full-VRAM fine-tunes push the card to its rail. Knowing the actual power profile by workload is the difference between sizing a 32 A UK feed correctly and tripping a breaker in production. The numbers below were captured on the gigagpu.com/ fleet across vLLM, SGLang, Diffusers, FLUX and Whisper pipelines. Order on the RTX 4090 24GB hosting page or browse all dedicated GPU options.

Contents

• 450 W TDP and the AD102 power profile
• Observed draw under inference workloads
• Tokens per watt, tokens per joule
• Undervolting and the efficiency frontier
• Rack and PSU density implications
• Monitoring power in production
• Production gotchas
• Verdict and tuning recipe

450 W TDP and the AD102 power profile

The 4090 reference design is rated 450 W board power (TGP), with a fast trip at ~600 W to protect the VRM and a slow trip on sustained over-spec. The 12VHPWR connector delivers up to 600 W. AD102 power scales near-linearly with frequency above 2.2 GHz and very steeply above 2.6 GHz; a 100 MHz reduction below stock typically removes 12-15 percent of the wattage with single-digit performance loss. That is the basis for the entire undervolt argument later in this article.

Inside the 450 W envelope, roughly 70 percent goes to the GPU core (SMs and tensor cores), 18 percent to GDDR6X (12 chips at 1.35 V running PAM4 at 21 Gbps), 8 percent to the VRM losses and 4 percent to the PCB rails (NVENC, NVDEC, display engines, fans). The memory rail rarely scales down because GDDR6X holds clock; it is one of the reasons idle-with-model-loaded sits around 25-28 W rather than the 13-22 W Windows desktop idle. On a server in steady-state inference, you can predict your draw from the workload profile far more accurately than from “TDP”.

Observed draw under inference workloads

Numbers below were captured with nvidia-smi --query-gpu=power.draw,utilization.gpu --format=csv -l 1 averaged over 60 seconds, on a card sitting at 27 C inlet temperature with the stock 450 W power limit. Real production traffic will skew toward the batched rows, not single-user.

Three observations are worth absorbing. Single-user LLM decode draws roughly 60 percent of TDP because tensor cores are mostly idle waiting for the bus; that is the same observation behind the low single-user TFLOPS utilisation number. Batched decode (16-32 requests) lifts draw to ~360 W because tensor cores fire on every token while the bus stays busy. Diffusion is the only workload that sustains 430-450 W routinely, because UNet steps are dense GEMM-heavy and the model fits in VRAM with constant high reuse. If you size your PSU and rack feed for “always 450 W”, you will overprovision by a third for an LLM-heavy fleet.

Tokens per watt, tokens per joule

Tokens per watt is the meaningful efficiency metric for an inference fleet because it directly maps to operating cost per output token. Tokens per joule (t/J) is the same quantity normalised to energy. Higher is better for both.

The three-and-a-bit tokens per joule peak on batched 8B FP8 is a useful planning number. At UK industrial electricity rates of around £0.20 per kWh (£5.5e-8 per joule), running at 3 t/J implies an electricity cost of about £0.018 per million output tokens – well under one percent of what a commercial API charges. The full economic picture including hosting, depreciation and amortised CapEx is in the monthly hosting cost piece, and the focused efficiency analysis lives at tokens per watt.

The lesson for product engineers: batch your traffic. Single-user decode wastes 75 percent of the wall energy on weight movement that produces only one token. Continuous batching in vLLM or TGI multiplies tokens per joule by 4-5x with no extra hardware.

Undervolting and the efficiency frontier

Two production methods on Linux:

# Method 1: hard cap on board power
sudo nvidia-smi -pl 360                     # set TGP cap to 360W
sudo nvidia-smi -q -d POWER | grep "Power Limit"  # verify

# Method 2: cap clock to silicon sweet spot
sudo nvidia-smi -lgc 0,2550                 # lock graphics clock to 2.55 GHz
# combine for predictable behaviour
sudo nvidia-smi -pl 360 -lgc 0,2550

# Reset to stock
sudo nvidia-smi -pl 450 -rgc

The power-limit method is preferred for fleets because it is transparent to the workload: the GPU manages clock down to fit the wall budget. The clock-cap method is preferred for single-tenant determinism because steady-state clock is fixed. Either way, thermal headroom improves in tandem – lower voltage means cooler junction means slower fans means quieter rack.

The efficiency peak sits between 320-360 W: roughly 7 percent throughput loss for 16 percent power savings, with junction temperatures dropping ~8 C. Below 280 W you start hurting tokens/joule because you're pushing into the bandwidth-bound regime where memory clock dominates and lower core clock just stretches kernel launch latency. The 380 W setting is the gigagpu.com/ fleet default for production hosts because it preserves 95 percent of throughput while extending VRM and capacitor life by a meaningful margin.

Rack and PSU density implications

The economics of a UK colocation are shaped by power density. Eight 4090s capped at 360 W draw 2,880 W on the GPUs alone, plus ~250 W for CPU, ~100 W for DDR5, ~60 W for NVMe and ~120 W for PSU losses at 92 percent efficiency – call it 3,400 W total. That fits comfortably under a 32 A 230 V UK feed (7,360 W) with margin for inrush. Stock 450 W on eight cards is 3,600 W of GPU plus 530 W of host plus PSU losses, ~4,500 W total – workable on a 32 A feed but margin is tight if any one card spikes briefly to 600 W. At 16 A (3,680 W), you cap at four 4090s undervolted to 380 W or three at stock.

For multi-card hosts, see the multi-card pairing guide for topology and the PCIe Gen 4 x16 piece for why tensor parallel is rarely the right answer on Ada.

Monitoring power in production

A minimal Prometheus exporter scrape pattern using nvidia_gpu_exporter:

# systemd unit excerpt
ExecStart=/usr/local/bin/nvidia_gpu_exporter \
  --query-gpu=power.draw,power.limit,temperature.gpu,temperature.memory,clocks.sm,clocks.mem,utilization.gpu,utilization.memory \
  --interval=5s \
  --listen-address=:9835

# PromQL alert: sustained over-budget power
- alert: GpuPowerOverBudget
  expr: avg_over_time(nvidia_gpu_power_draw_watts[5m]) > 400
  for: 10m
  labels: { severity: warning }
  annotations:
    summary: "GPU {{ $labels.uuid }} averaging > 400 W"

# PromQL: efficiency drift
- record: gpu:tokens_per_joule:5m
  expr: rate(vllm_generation_tokens_total[5m]) / rate(nvidia_gpu_power_draw_joules_total[5m])

The tokens-per-joule recording rule is the one to put on a dashboard: it instantly surfaces regressions caused by a kernel update, a thermal throttle event or a model swap. Pair it with the memory junction thermal alerts for a complete picture.

Production gotchas

1. 12VHPWR adapter cables can sag. Always use a native PSU cable; ATX 3.0 PSUs ship one. Adapter dongles have caused melting incidents at 450 W sustained.
2. Idle draw stays at 25-30 W with a model loaded. The memory clock holds high to keep weights warm. Don’t try to interpret 25 W as a problem.
3. Fan curve hysteresis matters. AIB cards have a 50 C zero-RPM threshold; under sustained inference they sit at 60+ C and fans never stop. That is correct behaviour.
4. Power-limit changes need persistence-mode on. nvidia-smi -pm 1 ensures the cap survives idle transitions; otherwise the cap can revert.
5. NVENC and NVDEC do not count against TGP in nvidia-smi. Heavy video pipelines can add 30-40 W invisible to the power.draw counter; budget separately. See NVENC AI pipelines.
6. Diffusion workloads spike 20 W above the LLM average. SDXL and FLUX are the worst-case for sustained TGP; size headroom for them.
7. Driver upgrades can shift the power curve by 5-8 percent. Validate efficiency after every CUDA bump in your CI suite.

Verdict and tuning recipe

For the typical gigagpu.com/ production deployment – 8B-class FP8 model, 8-32 concurrent users, 24/7 uptime – the right setup is:

• Power limit 380 W (preserves 95 percent throughput, gains 14 percent efficiency).
• Persistence mode on (nvidia-smi -pm 1).
• Memory junction alert at 95 C (long before throttle at 110 C).
• vLLM continuous batching with --max-num-seqs 32 for tokens-per-joule lift.
• Prometheus scrape at 5 s interval with the tokens-per-joule recording rule above.

For diffusion-heavy hosts, raise the cap to 420 W to absorb FLUX peaks. For rack densities above 4 cards per chassis, drop to 360 W and ensure inlet temperature stays under 27 C ASHRAE A1. For the cost and ROI side of the equation see the ROI analysis and monthly hosting cost pieces.

See also: tokens per watt deep-dive, thermal behaviour, monthly hosting cost, spec breakdown, multi-card pairing, vLLM setup, FP8 Llama deployment.