Mistral Nemo 12B with 128k context is distinctive on the RTX 5060 Ti 16GB at our hosting. Long-context workloads change the monthly math versus standard 7B class serving.
Contents
Throughput
Nemo 12B AWQ with FP8 KV cache:
- Batch 1 at 8k context: ~55 t/s
- Batch 8 at 8k: ~340 t/s aggregate
- Batch 4 at 32k context: ~210 t/s aggregate
- Batch 1 at 128k: ~38 t/s (single user)
Lower concurrency at long contexts because each sequence reserves more KV cache.
Monthly Capacity
Depends heavily on average context length. For 50% utilisation:
| Average Context | Output Tokens/Month |
|---|---|
| 8k (chat) | ~430M |
| 32k (RAG) | ~270M |
| 128k (long doc) | ~45M |
Long-Context API Math
Long-context API charges are dominated by input tokens. A single 128k query costs ~$0.09 on Mistral’s API at $0.7/M input. On dedicated hosting the marginal cost is zero.
For workloads with many long documents this is the main economic case for self-hosting – API costs scale with prompt length.
API Comparison
| API | Rate | Your Traffic Cost |
|---|---|---|
| Mistral Nemo API | ~$0.3 in / $0.7 out | Varies wildly with context length |
| Together Mistral Nemo | ~$0.3 blended | Varies |
For 500 long-document analyses/month at 64k input + 1k output, API cost: 500 × (64 × $0.30 + 1 × $0.70) / 1000 = ~$10. Low volume stays on API. High volume flips to dedicated.
Break-Even
For document-heavy workloads: dedicated typically wins above 5,000 long-document requests/month. Below that, API is cheaper. See break-even calculator.
Long-Context Fixed-Cost Hosting
128k Mistral Nemo on Blackwell 16GB. UK dedicated.
Order the RTX 5060 Ti 16GBSee also: deployment guide, long-context performance.
