Table of Contents
The data-flow audit is one of the things compliance teams ask for. This is the documented answer for self-hosted AI.
TL;DR
Self-hosted prompt flow: client → reverse proxy → LiteLLM → vLLM → memory → response. No third-party hop. No persistent storage by default. Compare to hosted API: prompt leaves your network, sits on US-resident provider for 7-30 days for "abuse review".
The flow
- Client sends prompt over TLS to reverse proxy (Caddy)
- Caddy validates auth and forwards to LiteLLM
- LiteLLM applies rate limit + budget check, forwards to vLLM
- vLLM tokenises, runs forward pass on GPU memory, returns tokens
- LiteLLM logs request metadata (no payload by default)
- Response returns through chain to client
- Memory wiped on subsequent batch eviction
Where data is NOT stored persistently: anywhere. By default, no logs of prompt content; memory is wiped on batch eviction.
Contrast with hosted APIs
- Hosted: prompt → US-based provider → buffered for 7-30 days → response
- Self-hosted: prompt → your server → response. Buffer = active inference only.
Verdict
Self-hosted data flow is dramatically simpler than hosted-API flow. Document it for your DPIA / privacy review.
Bottom line
Boring data flow = easier audit. See private AI hosting.
