Cost per 1M Tokens

We take the GPU’s estimated throughput in tokens per second, multiply by the number of seconds in a 30-day month (2,592,000), and divide the server’s monthly price (converted to USD) by that total. This gives the cost to generate one million tokens at maximum utilisation. Real-world costs will be higher if your server isn’t running inference around the clock.

The throughput and cost figures are benchmarked on LLaMA 3 8B at Q4_K_M quantisation. Larger models like 13B, 33B, or 70B will produce fewer tokens per second and therefore have a higher effective cost per million tokens. Smaller or more aggressively quantised models may be faster. Use the calculator above to estimate costs for your specific workload.

Not always. If your usage is very low — say a few thousand tokens per day — an API with pay-per-token billing may be cheaper because you’re not paying for idle server time. Self-hosting becomes more cost-effective as utilisation increases. As a rough guide, if you’re generating more than about 10–20 million tokens per month consistently, a dedicated GPU will typically save you money compared to flagship API models.

Proprietary frontier models like GPT-4o and Claude Opus are generally more capable on complex reasoning and creative tasks. However, open source models have closed the gap significantly — models like LLaMA 3, Mistral, Qwen, and DeepSeek perform extremely well for production use cases including chatbots, document summarisation, code generation, and RAG pipelines. For many workloads, the quality difference is negligible while the cost difference is enormous.

Yes. Tools like Ollama and vLLM support loading multiple models and switching between them. The constraint is VRAM — you need enough GPU memory to hold each active model. For example, a 24GB RTX 3090 can hold a 7B model and a smaller embedding model simultaneously. For multi-model production stacks, GPUs with 32GB+ VRAM like the RTX 5090 or RTX 6000 PRO are recommended.

Yes — most providers offer batch processing (typically 50% off) and prompt caching (up to 90% off repeated inputs). These can significantly reduce costs for eligible workloads. However, batch processing introduces latency (up to 24 hours for results), and caching only helps with repeated prompts. For real-time, high-volume inference with varied inputs, self-hosting remains substantially cheaper.

Most servers are provisioned within a few hours. Once you have SSH access, installing Ollama or vLLM takes a few minutes, and downloading a model is limited only by network speed. You can typically go from order to running inference within the same day.