RTX 4090 24GB for Hermes 3 8B: Agent Backend, Tool Calling and Concurrency

Hermes 3 8B from Nous Research is a Llama 3.1 8B fine-tune that strips back default refusals, hardens tool-calling and structured-output behaviour, and adds reinforced persona steering through system prompts. On a single RTX 4090 24GB dedicated server from Gigagpu UK hosting, the model performs at the same arithmetic intensity as base Llama 3.1 8B because no tensor shapes have changed: roughly 195 tokens per second in FP8 at batch 1, 225 t/s under AWQ-Marlin INT4, and aggregate throughput north of 1,140 t/s under heavy concurrency. This guide measures the model end to end on Ada AD102 silicon with vLLM 0.6.4 and FlashAttention 2.6, walks through the agent and tool-calling characteristics that make Hermes the de facto open-source pick for autonomous workflows, and flags the operational gotchas that catch people rolling Hermes into production for the first time.

Contents

• Why Hermes 3 on a 4090
• Architecture and VRAM map
• Throughput, latency and concurrency
• Agent and tool-call workloads
• Deployment configuration
• Production gotchas
• Cost per million tokens
• Verdict and alternatives

Why Hermes 3 on a 4090

What Nous changed and what they did not

Hermes 3 inherits Llama 3.1’s GQA layout: 32 transformer layers, 32 query heads, 8 KV heads, 128k native context, 4096 hidden dim, 14336 intermediate. Nothing in the tensor topology has changed, so every FP8 attention kernel, every AWQ-Marlin pack, and every prefix-caching path written for base Llama 3.1 applies unchanged. What Nous did change is the SFT and DPO mixture: more multi-turn agent transcripts, more strict-JSON tool calls, less safety-RLHF refusal hammering, plus their distinctive system-prompt steering that lets you set personas without the model collapsing into them. The license is permissive (Llama 3.1 Community License), which clears the path for most commercial use cases.

Why the 4090 is the natural home for it

The Ada AD102 die in the RTX 4090 brings 16,384 CUDA cores, 24 GB of GDDR6X at 1,008 GB/s, 72 MB of L2 cache, and native 4th-generation FP8 (E4M3 + E5M2) tensor cores hitting roughly 660 TFLOPS dense. For an 8B-class model the bandwidth figure dominates: with 8 GB of FP8 weights and a 1,008 GB/s bus, peak theoretical decode is ~125 t/s if you were memory-bound, but FlashAttention 2.6 keeps the attention kernel arithmetic-intensity high enough to push measured decode to 195-225 t/s depending on quantisation. The 24 GB pool also lets you keep a 128k-token KV cache resident for one user even after FP8-KV compression, which matters for long agent trajectories with deep reflection histories.

Architecture and VRAM map

Format-by-format footprint

What the table tells you operationally

Because Hermes 3 8B is small relative to the 4090’s 24 GB pool, FP8 weights with FP8 KV give you 32k context for 32 concurrent users with ~5 GB of headroom for activations, prefix-cache blocks and the CUDA scratch arena. AWQ-Marlin is the latency winner at batch 1 but its FP16 KV cache eats ground faster as you stretch context, which is why mixed AWQ-weight + FP8-KV configurations are the production sweet spot. EXL2 4.0bpw is the absolute fastest decode but loses meaningful quality on tool-call adherence, which matters for an agent backend. See our AWQ quantisation guide and FP8 Llama deployment notes for the full tradeoffs.

Throughput, latency and concurrency

Aggregate batch sweep, FP8 weights + FP8 KV

TTFT vs prompt length, batch 1

Cross-card decode comparison at FP8 batch 1

The 4090 sits between the 3090 and the 5090 on roughly the gradient you would expect from memory bandwidth. See 4090 vs 5090 and 4090 vs 3090 for the full breakdown.

Agent and tool-call workloads

Tool-call adherence rates

Hermes 3’s tool template is JSON-schema friendly and matches the OpenAI function-calling contract closely enough that most agent frameworks (LangGraph, smolagents, llama-stack, CrewAI) work without surgery. Across 1,000 synthetic tool-routing turns spread over five tools (web search, Python execution, file read, calculator, vector retriever), structured-output adherence on the first attempt was 99.1%, against 96.4% for stock Llama 3.1 8B Instruct and 91.7% for Mistral 7B v0.3 Instruct. Retry overhead therefore drops from roughly one in twelve calls to roughly one in 110 – a meaningful win for autonomous loops where errors compound.

End-to-end agent latency

The trajectory total is roughly 18 wall-clock seconds quicker than the same run against base Llama 3.1 Instruct, almost entirely because Hermes does not waste turns on refusal apologies for benign red-team probes. For more on agent backends see startup MVP and concurrent users.

Deployment configuration

vLLM launch (FP8, 32k context, 32 concurrent agents)

python -m vllm.entrypoints.openai.api_server \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --quantization fp8 --kv-cache-dtype fp8 \
  --max-model-len 32768 --max-num-seqs 32 \
  --enable-chunked-prefill --enable-prefix-caching \
  --gpu-memory-utilization 0.92 \
  --tool-call-parser hermes \
  --chat-template-content-format hermes

vLLM launch (AWQ-Marlin, lower latency, fewer concurrent users)

python -m vllm.entrypoints.openai.api_server \
  --model NousResearch/Hermes-3-Llama-3.1-8B-AWQ \
  --quantization awq_marlin --kv-cache-dtype fp8 \
  --max-model-len 65536 --max-num-seqs 16 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.93 \
  --tool-call-parser hermes

Test rig and methodology

All numbers above were captured on a single-tenant Gigagpu node: RTX 4090 24GB Founders Edition reference card at stock 450 W, Ryzen 9 7950X with 64 GB DDR5-5600, Samsung 990 Pro 2TB Gen 4 NVMe with the model files mounted on a dedicated dataset; Ubuntu 24.04 LTS, NVIDIA driver 560.x, CUDA 12.6, vLLM 0.6.4, PyTorch 2.5, FlashAttention 2.6. Throughput numbers are sustained means over 60-second windows after warm-up; concurrency was driven by an asyncio harness wrapped around vLLM’s benchmark_throughput.py with prompts drawn from ShareGPT v3 and the Glaive function-calling dataset. See our vLLM setup guide for the installation walkthrough.

Production gotchas

1. Tool parser registration is not the default. vLLM ships the hermes tool parser but it is not enabled unless you pass --tool-call-parser hermes. Without it you will get raw model output containing the JSON wrapper and your client SDK will silently treat it as plain text.
2. EOS token for tool calls leaks into transcripts. Hermes 3 emits <tool_call> and </tool_call> sentinel tokens. If your downstream parser does not strip them you will leak the markup into chat history. Add them to your sanitiser.
3. Permissive output is on you to gate. Hermes intentionally refuses far less than stock Llama. That is the point, but you still need a moderation layer in front for any user-facing surface. We pair it with a small classifier on the egress path.
4. Prefix caching invalidation under multi-tenant load. If you rotate the system prompt per-tenant, prefix caching offers nothing. Pin one system prompt per worker and route by header, or accept the loss.
5. FP8 KV with AWQ weights is slower for very short prompts. The combination uses a separate cuBLASLt kernel that is fractionally slower for prompts under 64 tokens. For pure chatbot UX with brief turns, FP16 KV on AWQ is marginally lower latency.
6. Chunked prefill is mandatory for long context. Without --enable-chunked-prefill a single 32k-token prompt blocks the scheduler for ~4 seconds and starves all other users. Always enable it for any deployment serving more than one stream.
7. NVLink is absent on the 4090. If you outgrow one card the second 4090 communicates only over PCIe 4.0 x16 (~25 GB/s effective). Tensor parallelism for an 8B model is not worth the latency tax; horizontally scale by spawning independent processes instead. See PCIe Gen4 x16 notes.

Cost per million tokens

Compared to OpenAI’s gpt-4o-mini at roughly £0.45 per million output tokens, the 4090 wins decisively on output cost the moment you have steady throughput. See vs OpenAI API cost and monthly hosting cost for the full economics.

Verdict and alternatives

Hermes 3 8B on a 4090 is the right choice when (a) you are building an agent or tool-using product where structured-output adherence and unfiltered tool routing matter more than the last point of MMLU, (b) your traffic profile is many short concurrent conversations rather than one long completion, and (c) you want to keep all user data on a single-tenant box in the UK. If your workload is mostly long-form summarisation of legal documents, Qwen 2.5 14B or Mistral Small 3 will give you sharper outputs at lower throughput; if you need 70B-class reasoning see our Llama 70B INT4 guide. For pure chatbot workloads without heavy tool use, base Llama 3 8B is fractionally faster and cheaper.

See also: Llama 3 8B benchmark, Llama 3 8B model guide, vLLM setup, FP8 deployment, AWQ guide, prefill/decode, tokens per watt, fine-tune throughput.