Cost of Serving Llama 3 70B AWQ on RTX 4090 24GB: Capacity, $/M and Break-Even

Serving a 70B model on a single GPU used to mean an A100 80 GB at $2-3 per hour. Thanks to AWQ INT4 quantisation and vLLM’s marlin kernels, the RTX 4090 24GB dedicated server can host Llama 3.1 70B comfortably and serve it at a price-per-million-tokens that demolishes hosted APIs above modest volumes. This article works the numbers from monthly fee through capacity to effective $/M token, with MAU break-even tables and a 12-month TCO comparison on GigaGPU dedicated hosting.

Contents

• Llama 3.1 70B AWQ on a 4090
• vLLM launch and tuning
• Monthly fixed cost
• Throughput in production
• Tokens per month and volume tables
• Effective $/M token and MAU break-even
• 12-month TCO vs hosted APIs
• Production gotchas

Llama 3.1 70B AWQ on a 4090

AWQ packs the 70B weights at 4 bits per weight, around 18.9 GB resident. Add KV cache, vLLM scheduler overhead and CUDA context: ~22-23 GB on a 24 GB card. With --gpu-memory-utilization 0.95 and a 16k-token context cap, vLLM serves 4 concurrent streams comfortably with KV-cache headroom for prefill bursts.

For the deployment recipe see the Llama 70B INT4 deployment guide; for the underlying benchmark numbers see 70B INT4 benchmark.

vLLM launch and tuning

The canonical launch line, with FP8 KV cache to claw back VRAM:

python -m vllm.entrypoints.openai.api_server \
  --model hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4 \
  --quantization awq_marlin --kv-cache-dtype fp8 \
  --max-model-len 16384 --max-num-seqs 4 \
  --gpu-memory-utilization 0.95

Key choices: awq_marlin picks the optimised Ada kernel; kv-cache-dtype fp8 halves KV memory at no measurable accuracy cost on Ada; max-num-seqs 4 matches the achievable concurrency given KV constraints; gpu-memory-utilization 0.95 leaves just enough headroom for CUDA context. For larger batches drop max-model-len to 8192.

Monthly fixed cost

For modelling we will use $700/month cash outlay and $1,150/month loaded cost as midpoints.

Throughput in production

vLLM 0.6 with continuous batching produces these numbers on the 4090:

Batch 4-6 is the sweet spot, around 72-90 aggregate t/s. We model on 80 t/s sustained for the rest of the article. For raw concurrency see concurrent users.

Tokens per month and volume tables

How long does it take a 4090 to serve different volume targets?

The 70B model on a single 4090 hits its capacity ceiling around 200 M tokens/month. For higher volumes either run smaller models (Qwen 32B does 654 M/month, see Qwen 32B cost) or scale horizontally across multiple cards via multi-card pairing.

Effective $/M token and MAU break-even

MAU break-even, assuming 25k tokens per active user per month (typical chat/RAG):

12-month TCO vs hosted APIs

Hosted Llama 3.1 70B (Together, Fireworks, DeepInfra) averages $0.85-0.90 per million tokens blended. Compare 12-month TCO at three volume points:

The 70B-on-4090 economics work best when you compare against premium frontier APIs (GPT-4o, Sonnet, Opus), not against hosted Llama. For absolute cheapest cents-per-token Llama, hosted Together is hard to beat. For everything else, dedicated wins.

Production gotchas

1. KV cache OOM on long prompts: a 70B with 16k context can consume 8+ GB of KV per stream. Cap max-model-len aggressively or you OOM under burst.
2. FP8 KV cache rounding: at very long contexts (>12k) FP8 KV starts to drift on attention scores. For long-context legal/medical workloads use BF16 KV at the cost of half the concurrency.
3. Together rate limits: hosted 70B has per-org QPS caps; if your spike exceeds, you get 429s. Self-hosted has no such cap until VRAM saturates.
4. AWQ accuracy regression: INT4 loses 1-2 points on MMLU vs BF16. For most chat that is invisible; for hard reasoning eval at scale, consider FP8 deployment on a 5090 or H100 instead.
5. vLLM updates breaking AWQ: vLLM is on a fast release cadence and AWQ support has had churn. Pin vllm==0.6.3 and test before upgrading.
6. Cold-start cost: 70B AWQ takes ~90 seconds to load. Persistence mode and a warm pool process matter; do not cycle the server casually.
7. Power and thermals at 4 streams sustained: 430-440 W draw for hours; verify thermal performance ahead of multi-day runs.

Verdict

Self-hosted Llama 3.1 70B AWQ on a single 4090 lands at $3.74-6.67 per million tokens at 70-90% utilisation. That beats GPT-4o ($5) and Claude Sonnet ($7) on cash $/M, and it is significantly cheaper at frontier-quality output than per-token APIs once you exceed roughly 100-150 M tokens per month. For raw cheapest-Llama, hosted Together is unbeatable; for privacy, UK residency, custom adapters and predictable bills, dedicated 4090 wins. Cross-check the case with the vs OpenAI, vs Anthropic and break-even calculator.

See also: 70B INT4 deployment, 70B INT4 benchmark, AWQ guide, vs OpenAI, vs Anthropic, vs Together AI, break-even, monthly hosting cost, 4090 for Llama 70B.