Benchmark Overview
Model loading time is your cold start penalty. Every server restart, model swap, or scaling event requires loading model weights from disk into GPU memory. For a 70B INT4 model, this can range from 8 seconds on NVMe to over 3 minutes on HDD. We benchmarked loading times across storage types and GPU configurations on dedicated GPU hosting to quantify the cold start cost.
Test Configuration
Models: Llama 3 8B FP16 (16GB), Llama 3 8B INT4 (5GB), Llama 3 70B INT4 (38GB), Llama 3 70B FP16 (140GB, 2 GPUs). Storage: NVMe Gen 4 (7,000 MB/s read), SATA SSD (550 MB/s read), HDD (200 MB/s read). Served via vLLM and Ollama. Measured from process start to first request served.
Model Loading Times
| Model | Size on Disk | NVMe Gen 4 | SATA SSD | HDD |
|---|---|---|---|---|
| Llama 3 8B INT4 | 5 GB | 3s | 12s | 28s |
| Llama 3 8B FP16 | 16 GB | 6s | 32s | 85s |
| Llama 3 70B INT4 | 38 GB | 10s | 72s | 195s |
| Llama 3 70B FP16 (2 GPUs) | 140 GB | 28s | 260s | 720s |
| Mixtral 8x7B INT4 | 26 GB | 8s | 50s | 135s |
Storage Impact Analysis
NVMe Gen 4 storage loads models 5-10x faster than SATA SSD and 20-30x faster than HDD. The 70B INT4 model loads in 10 seconds on NVMe versus 72 seconds on SATA. This difference matters for cold starts, model swapping, and recovery from crashes. NVMe is essential for production deployments where downtime must be minimised. Refer to the vLLM production guide for storage configuration.
PCIe bandwidth between storage and GPU also affects loading. PCIe Gen 4 x16 provides 32 GB/s, which exceeds NVMe read speed. The storage read speed is the bottleneck, not the PCIe bus. See GPU specifications for PCIe compatibility data.
Engine Loading Differences
vLLM adds 2-5 seconds of initialisation overhead beyond raw model loading for setting up PagedAttention memory pools and warming the CUDA context. Ollama uses memory-mapped loading (mmap) which shows the model as “loaded” faster but may incur latency on first inference as pages fault from disk. Compare engines at vLLM vs Ollama. Check token benchmarks for inference performance after loading.
Recommendations
Always use NVMe storage for production LLM serving. The cost difference between NVMe and SATA is negligible compared to the cold start improvement. Keep frequently used models on NVMe and archive infrequently used models on cheaper storage. Deploy on GigaGPU dedicated servers with NVMe Gen 4 storage and private hosting. Visit the benchmarks section and LLM hosting guide for more deployment data.
