RTX 4090 24GB Concurrent Chat Users: Capacity Benchmark

The most common sizing question we get is “how many users can a single RTX 4090 24GB handle?” The honest answer depends on the model, the SLA you set for first token and inter-token latency, the prompt length distribution, and how chatty your audience actually is. This benchmark establishes a per-model active-user envelope at a chat-grade SLA (TTFT p95 below 800 ms, decode at least 30 t/s/user), translates active concurrency to monthly active users (MAU) with a defensible formula, and documents the seven operational gotchas that wreck capacity in production. All numbers come from a stock UK dedicated 4090 running vLLM 0.6.4 with continuous batching and FP8 KV cache where supported.

Contents

• Defining the SLA
• Methodology and load generator
• Per-model concurrency table
• From active to MAU
• Prefix caching boost
• Handling bursts and queue admission
• When to add a second card
• Production gotchas

Defining the SLA

“Concurrent users” is meaningless without a target. We use a chat-style SLA modelled on what users actually tolerate in production:

• TTFT (time-to-first-token) p95 below 800 ms. Above this users perceive the system as “thinking” rather than “answering” and abandon the turn.
• Inter-token latency below 33 ms median (~30 t/s perceived per user). Faster than human reading speed; slower feels stuttery.
• Hard ceiling: TTFT p99 below 2,000 ms. Beyond this, the load balancer should return 503 to protect in-flight requests.

Prompts assumed at 600 input tokens (a system message, a few-shot exemplar, the user turn) with 250 output tokens. This is representative of mainstream chat and SaaS RAG workloads; for coding agents (longer context) or voice (shorter both ways) the envelope shifts and we cover those in the coding assistant and voice assistant pieces.

Methodology and load generator

Load is generated by a Python harness that maintains N persistent OpenAI-format streaming sessions, each issuing a new turn within 200 ms of the previous reply completing (mimicking active typing). Sessions are warmed for 60 s before measurement begins; we then run for 600 s and record per-request TTFT and per-token latency. Test rig is the standard 4090 host: Ryzen 9 7950X, 64 GB DDR5-5600, Samsung 990 Pro Gen 4 NVMe, Ubuntu 24.04, driver 560.x, CUDA 12.6. vLLM 0.6.4 with PyTorch 2.5 and FlashAttention 2.6.

# Standard launch (Llama 3 8B FP8 with FP8 KV)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --quantization fp8 --kv-cache-dtype fp8 \
  --max-model-len 65536 --max-num-seqs 32 \
  --enable-chunked-prefill --enable-prefix-caching \
  --gpu-memory-utilization 0.92

The exact tuning parameters that matter most for the numbers below: --max-num-seqs sets the in-flight batch ceiling; --enable-chunked-prefill stops a long incoming prompt from stalling decodes for in-flight users; --enable-prefix-caching reuses KV for stable system prompts; --kv-cache-dtype fp8 halves KV memory and almost halves bandwidth pressure on decode.

Per-model concurrency table

“Active users” means users whose request is currently in flight or whose turn is queued less than one second. The 4090 sustains the following inside the SLA above:

Llama 3 8B FP8 is the workhorse: 30 active users, comfortable inside the SLA, with several gigabytes of VRAM free for KV cache headroom. Mistral 7B is slightly faster (smaller model, fewer parameters per layer) at 32 active. Qwen 14B AWQ trades half the throughput for measurably better instruction following. Llama 70B AWQ INT4 is the quality flagship but only suits 3-4 active users on a single 4090; for multi-user 70B production, plan for tensor parallelism over two cards as documented in the Llama 70B INT4 deployment guide.

Why Phi-3 mini holds 80 active

Phi-3 mini at 3.8B parameters fits substantially in the 4090’s 72 MB L2 cache and runs decode at ~480 t/s single-stream. Aggregate at batch 80 reaches ~1,800 t/s, more than enough to sustain 30 t/s per user. It is the right choice for routing, classification, and high-volume semi-structured tasks where you do not need flagship quality.

From active to MAU

Active concurrency is what the GPU sees. MAU is what your product manager cares about. The conversion uses three multipliers, all of which you should calibrate to your audience over the first month of production data:

• Hourly active fraction (h). Fraction of MAU active in a given hour. Typical SaaS: 0.05-0.15. B2B 9-5 tools: 0.20-0.40 in working hours.
• Talking time fraction (t). Fraction of an active hour the user is actually waiting on a response. Typical chat: 0.20-0.40.
• Burst factor (b). Ratio of peak hour to average hour. Typical SaaS: 2-3x.

The formula: MAU = active_users * 1 / (h * t * b). For Llama 3 8B FP8 with 30 active, h=0.10, t=0.30, b=2.5, the result is roughly 4,000 MAU. If your audience is heavier (h=0.20, t=0.40, b=2), it drops to 1,875 MAU. The table below shows a few sensible operating points:

The “light SaaS” column assumes h=0.05, t=0.20, b=3, typical of consumer products with broad reach and low engagement. The “heavy B2B” column assumes h=0.30, t=0.40, b=1.5, typical of internal tools used as part of a workflow.

Prefix caching boost

If your system prompt and few-shot are fixed (most chatbots), enable vLLM’s --enable-prefix-caching. We see a 1.6-2.0x increase in sustainable concurrent users for the same SLA, because the per-request prefill collapses for cached prefixes. With a 1500-token system prompt, cold prefill takes 140 ms; warm reuses the cache and hits 15 ms. For a service running at 30 active users, that frees roughly 3.5 GPU-seconds per second of wall time, available either for more users or for tighter TTFT. See the prefill and decode benchmark for the underlying microbenchmark.

When prefix caching does not help

Prefix caching is a no-op when each request has a unique prefix (e.g., per-user RAG context concatenated before the system prompt). The fix is to put RAG content after the system prompt so the system prefix remains cacheable; details in the vLLM setup guide.

Handling bursts and queue admission

Set queue admission to 1.5x the steady-state SLA capacity, with a hard cap. Beyond that, return 503s to load balancers; vLLM will protect existing in-flight requests from latency collapse. The right operational pattern uses two metrics: vllm:num_requests_running and vllm:num_requests_waiting. If running > max_num_seqs and waiting > 1.5 x max_num_seqs, return 503 from your gateway. We cover the full pattern in the vLLM setup guide and the FP8 Llama deployment piece.

When to add a second card

For Llama 3 8B FP8, two 4090s behind a round-robin load balancer linearly scales to 60 active users; vLLM has no inter-replica state and the gateway can be a simple least-loaded HTTP balancer. For Llama 70B you are better off with tensor parallelism over two cards (TP=2), which roughly doubles single-stream decode and pushes active users from 3-4 to 8-10. For deployment specifics see our Llama 70B INT4 deployment guide and the multi-card pairing piece.

Hybrid model strategies

One pattern that consistently outperforms a single-model deployment is routing 80% of traffic to Llama 3 8B FP8 and the remainder to Qwen 14B AWQ on the same card by reserving 6 GB of VRAM for the secondary model. A simple classifier (DeBERTa or even a heuristic) routes “easy” turns to the fast model and “hard” turns to the better one. Effective concurrency goes up, perceived quality stays high. The multi-tenant SaaS guide documents the pattern.

Production gotchas

• Prompts longer than max_num_batched_tokens stall the queue. If a 16,000-token RAG prompt arrives with --max-num-batched-tokens 4096, chunked prefill divides it across iterations, which is fine; without chunked prefill, the in-flight batch hangs for hundreds of milliseconds. Always enable chunked prefill.
• KV cache fragmentation eats capacity over time. After 24 hours of mixed sequence lengths, vLLM’s PagedAttention can fragment enough to cost 5-10% of effective batch. Restart the worker daily during low traffic, or use the eviction tuning in --swap-space.
• Long-tail prompts dominate p99 TTFT. If 1% of your prompts are 16k tokens, your p99 TTFT will be dominated by their prefill cost, not by your steady-state. Bucket and reject pathological prompts at the gateway.
• Streaming is non-negotiable for perceived latency. A 250-token reply at 30 t/s takes 8 seconds; without streaming the user waits 8 seconds for a single block. With streaming they read along. Always stream.
• Don’t trust max_num_seqs blindly. The number you can actually serve is gated by KV cache memory and prompt-length distribution, not by the flag. Measure under realistic load and tune downward, never upward.
• Model warm-up is not free. First request after cold start takes 2-4 seconds for engine compile. Issue 5-10 warm-up requests at deploy time before flipping the load-balancer to ready.
• Watch for chat history bloat. If your client sends the full conversation history every turn, prompt length grows monotonically and active capacity falls over a session. Use server-side conversation compaction or a sliding window.

Verdict: sizing the 4090 to your audience

For most B2B chat products, one 4090 running Llama 3 8B FP8 supports a few thousand MAU with comfortable headroom; for high-volume consumer chat the same card supports tens of thousands when usage is uncorrelated. For quality-sensitive applications (legal, medical, technical Q&A) Qwen 14B or Llama 70B is the better choice at lower density. The right answer is almost never “scale the largest model”; it is “route by query difficulty and pay for the model only when needed”. For broader sizing, see the Llama 3 8B benchmark, the Qwen 14B benchmark, and the cost-vs-OpenAI analysis. For smaller audiences the 5060 Ti concurrent users writeup is the right comparator.

See also: chatbot backend, prefill and decode, Llama 3 8B benchmark, vLLM setup, FP8 Llama deployment, tokens per watt, multi-tenant SaaS.