A fintech company reduced their monthly AI inference bill from $4,200 to $890 by implementing 8 of the 20 optimisations in this checklist — a 79% reduction without any degradation in output quality. Most teams leave 40-70% of potential savings on the table because they optimise the model but ignore the serving stack, or optimise throughput but ignore prompt efficiency.
Model Selection Optimisations
1. Right-size your model. A 70B model is overkill for 80% of production use cases. Benchmark your specific task on 7B, 13B, and 70B models. If the 7B model achieves 90%+ of the 70B’s quality on your evaluation set, the cost difference is 5-10x. Check the cheapest GPU for inference by model size.
2. Fine-tune small instead of prompting large. A fine-tuned 7B model often outperforms a prompted 70B model on domain-specific tasks. The fine-tuning cost is a one-time investment; the inference savings compound monthly.
3. Use specialist models for specialist tasks. Do not route OCR, embedding, classification, and generation through the same large model. Smaller task-specific models (BGE for embeddings, Surya for OCR, a classifier head for routing) are orders of magnitude cheaper per operation.
4. Evaluate newer models quarterly. A model released 6 months after your current one may deliver the same quality at half the parameter count. Mistral 7B outperforms Llama 2 70B on many benchmarks at 10x lower cost.
Quantisation and Compression
5. Quantise to 4-bit (AWQ or GPTQ). 4-bit quantisation reduces VRAM usage by 75% and increases throughput by 40-60% with less than 2% quality degradation on most tasks. This single optimisation can halve your GPU requirements.
6. Use GGUF for CPU offloading. If a model barely exceeds your GPU VRAM, GGUF format allows partial CPU offloading — keeping hot layers on GPU while cold layers use system RAM. Slower than full GPU inference but cheaper than upgrading to a larger GPU.
7. Apply speculative decoding. Use a small draft model (1-3B parameters) to generate candidate tokens verified by your main model. Throughput improvements of 2-3x with identical output quality.
Serving Stack Optimisations
8. Deploy with vLLM or TensorRT-LLM. vLLM’s PagedAttention and continuous batching increase throughput by 3-5x over naive PyTorch inference. This is the single highest-impact serving optimisation.
9. Enable continuous batching. Sequential request processing wastes GPU cycles. Continuous batching fills idle compute slots with new requests while previous ones are still generating, maximising utilisation.
10. Set appropriate max_tokens limits. Default max_tokens is often 2048 or 4096. If your average response is 200 tokens, setting max_tokens to 500 reduces KV-cache memory waste and allows higher concurrency.
11. Implement request queuing with priority. A queue system prevents GPU overload during traffic spikes and allows prioritisation of latency-sensitive requests over batch jobs.
Prompt and Pipeline Optimisations
12. Compress system prompts. A 2,000-token system prompt processed with every request costs as much as the actual user query. Reduce system prompts to essential instructions. At 100K queries/day, trimming 500 tokens from the system prompt saves significant cost per million tokens.
13. Cache common responses. If 20% of queries are near-duplicates (common in customer support), implement semantic caching. Hash the query embedding and return cached responses for queries above a similarity threshold.
14. Use RAG context windows efficiently. Retrieve 3 chunks instead of 10. Most RAG quality comes from the top 2-3 results. Reducing context window size from 4,000 to 1,500 tokens cuts generation cost by 60%.
15. Implement tiered routing. Route simple queries to a 7B model and complex queries to a 70B model. A lightweight classifier (or even regex rules) can route 60-70% of traffic to the cheaper model.
Infrastructure Optimisations
16. Consolidate models on fewer GPUs. Run embeddings, reranking, and small classification models on the same GPU as your LLM if VRAM allows. One GPU at 80% utilisation is cheaper than two at 40%.
17. Schedule batch jobs during off-peak hours. If your GPU handles real-time inference during business hours and batch processing at night, you get two workloads for the price of one GPU.
18. Monitor and act on utilisation metrics. A GPU running at 25% utilisation is 4x more expensive per query than one at 100%. If utilisation is consistently below 40%, downgrade to a cheaper GPU. The GPU vs API comparison helps determine if low-utilisation self-hosting still beats API pricing.
19. Use annual contracts for stable workloads. Annual GPU hosting contracts save 15-25% over monthly billing. On an RTX 6000 Pro, that is $960-$1,260 per year.
20. Move off APIs to self-hosted. The largest single cost reduction. Self-hosting on dedicated GPU infrastructure is 5-20x cheaper than API providers at any meaningful volume. The break-even analysis quantifies exactly when the switch pays off.
Implement These Optimisations on GigaGPU
Every optimisation in this checklist works best on dedicated GPU hosting where you have full control over the serving stack, model configuration, and resource allocation. GigaGPU’s open-source LLM hosting comes pre-configured with vLLM and quantisation support, so you can start optimising immediately.
Calculate your potential savings with the LLM cost calculator, or explore private AI hosting for optimised deployments with compliance requirements. More cost reduction strategies on the cost blog.
