Table of Contents
vLLM defaults are tuned for benchmarks, not 16 GB cards. This page is the actual launch config we use on 5060 Ti customers.
TL;DR
Optimal 5060 Ti vLLM config: FP8 weights, FP8 KV cache, max-num-seqs=48, max-model-len=16384, gpu-memory-utilization=0.92, prefix caching enabled. Sustains ~880 tok/s on Mistral 7B with comfortable headroom.
Install
sudo apt install -y python3.10-venv
python3.10 -m venv ~/vllm && source ~/vllm/bin/activate
pip install vllm==0.6.3Optimal launch config
vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
--host 0.0.0.0 --port 8000 \
--quantization fp8 \
--kv-cache-dtype fp8_e4m3 \
--max-model-len 16384 \
--max-num-seqs 48 \
--gpu-memory-utilization 0.92 \
--enable-prefix-caching \
--served-model-name mistral-7b \
--api-key sk-internal-tokenWhat each flag does
--quantization fp8: Blackwell native FP8 path. ~50% throughput vs FP16.--kv-cache-dtype fp8_e4m3: halves KV cache memory. Critical on 16 GB.--max-model-len 16384: caps context. 16K is plenty for most chatbots; 32K eats too much KV.--max-num-seqs 48: bound concurrent batches. Prevents OOM under load spikes.--gpu-memory-utilization 0.92: leave 8% headroom for unaccounted activations.--enable-prefix-caching: free 30-50% throughput on chat workloads.
Verdict
The config above is the right starting point for any 7B-8B model on a 5060 Ti. Adjust max-model-len and max-num-seqs based on your specific traffic profile.
Bottom line
This is the config we ship on customer 5060 Ti deployments. See batch size tuning for the deeper dive.
