Table of Contents
Time-to-first-token (TTFT) is the metric users actually feel. Median TTFT under 500 ms, p99 under 1 s — that’s the bar for a chatbot to feel snappy. On a 5060 Ti 16 GB hitting that bar takes work; this page documents what we tune in customer deployments.
The biggest wins on a 5060 Ti TTFT p99: FP8 weights, prefix caching, FP8 KV cache, tight max-num-seqs, warm-up requests on startup, and speculative decoding. Each is worth 50–200 ms of headline latency.
Why TTFT p99 matters
Users tolerate variance better than they tolerate slowness. A chatbot that feels fast 99% of the time and stalls 1% of the time loses fewer users than one that’s consistently mediocre. P99 is what catches the slowness; that’s what to optimise.
Baseline TTFT on the 5060 Ti
Mistral 7B FP16, vLLM defaults, 50 concurrent users, 1K-token prompt:
- Median TTFT: ~280 ms
- p99 TTFT: ~720 ms
- Aggregate throughput: 580 tok/s
That’s the "works out of the box" baseline. Six things move it.
Six knobs that actually move the number
1. FP8 weights (Blackwell native)
Halves prefill computation. Median TTFT drops to ~180 ms; p99 to ~480 ms. The single biggest knob on Blackwell hardware.
2. Prefix caching
vLLM’s --enable-prefix-caching caches the KV state of repeated prefixes. For chat workloads with shared system prompts, hits are 80%+. Median TTFT drops by another ~80 ms.
3. FP8 KV cache
--kv-cache-dtype fp8_e4m3 halves cache memory, lets you fit more concurrent requests in the prefill pool. Reduces queueing under load. p99 TTFT drops by ~150 ms at 50+ concurrent users.
4. Tight --max-num-seqs
vLLM’s default is 256 concurrent sequences. On a 16 GB card that’s optimistic — the KV pool fills before that bound. Set it to 32–64 explicitly. Reduces tail latency by ~100 ms at p99.
5. Warm-up requests on startup
First request after vLLM starts hits cold CUDA kernels and takes 1–3 s. Send a synthetic warmup request before traffic arrives. Eliminates the first-request spike.
6. Speculative decoding
Pair with a 1B draft model. Reduces TPOT by ~40% on chat workloads. Doesn’t directly affect TTFT but reduces total response time and frees the GPU for the next prefill.
Two things that do not help (despite advice you will read)
- –max-model-len 8192 — restricting context length doesn’t reduce TTFT for short prompts. The KV cache is allocated lazily.
- Higher GPU clocks — Blackwell’s boost clock is dynamic and CUDA-bound, not user-tunable. Stock cooling is fine.
Verdict
After applying all six knobs on a 5060 Ti running Mistral 7B:
- Median TTFT: ~180 ms (was 280)
- p99 TTFT: ~420 ms (was 720)
- Aggregate throughput: ~1,100 tok/s (was 580)
That’s a real chatbot SLA — comfortable median, p99 under half a second, throughput nearly doubled. All free.
Bottom line
If your 5060 Ti chatbot feels slow, work through these six knobs in order. FP8 + prefix caching alone usually halves TTFT. After that, speculative decoding is the highest-leverage move — see our speculative decoding guide.
