What You’ll Build
In a single afternoon, you will have a multi-tenant chatbot platform where each client gets their own branded AI assistant backed by isolated knowledge bases, custom system prompts, and per-tenant usage tracking. One GPU server supports 20-50 concurrent tenants sharing the same model while maintaining strict data separation. This is the foundation for a chatbot-as-a-service business running on dedicated GPU hosting.
Agencies, SaaS companies, and AI consultancies need a way to deploy chatbots for multiple clients without spinning up separate infrastructure per customer. A multi-tenant architecture on open-source LLMs eliminates per-seat licensing from commercial providers and gives you full control over pricing, features, and data residency for each tenant.
Architecture Overview
The platform has four layers: an API gateway with tenant routing, a shared vLLM inference backend, per-tenant RAG vector stores, and a management dashboard for onboarding and monitoring. Incoming requests include a tenant API key that the gateway validates and uses to load the correct system prompt, knowledge base namespace, and rate limits.
The LLM model is loaded once in GPU memory and shared across all tenants. Tenant isolation happens at the prompt and retrieval layer, not the model layer. Each tenant’s documents are indexed in a separate collection within the vector database. LangChain handles the per-request assembly of system prompt, RAG context, and conversation history. This shared-model architecture keeps GPU utilisation high and cost per tenant low.
GPU Requirements
| Tenant Count | Recommended GPU | VRAM | Concurrent Users |
|---|---|---|---|
| Up to 10 tenants | RTX 5090 | 24 GB | ~30 concurrent |
| 10 – 30 tenants | RTX 6000 Pro | 40 GB | ~80 concurrent |
| 30 – 100 tenants | RTX 6000 Pro 96 GB | 80 GB | ~200 concurrent |
Concurrent user count matters more than tenant count since most tenants have sporadic usage. vLLM’s continuous batching efficiently multiplexes requests from different tenants into the same inference batch. Read our self-hosted LLM guide for GPU memory planning across tenant volumes.
Step-by-Step Build
Deploy vLLM on your GPU server with the OpenAI-compatible API enabled. Set up PostgreSQL for tenant metadata, API keys, and usage logs. Configure a vector database with namespace support for per-tenant document isolation. Build the API gateway using FastAPI with middleware for tenant authentication and rate limiting.
# Tenant-aware request handler
async def chat(request: ChatRequest, tenant: Tenant):
# Load tenant-specific configuration
system_prompt = tenant.system_prompt
rag_context = await vector_db.query(
collection=f"tenant_{tenant.id}",
query=request.message, top_k=5
)
# Assemble prompt and call shared vLLM
messages = [
{"role": "system", "content": system_prompt},
{"role": "system", "content": f"Context: {rag_context}"},
*request.history,
{"role": "user", "content": request.message}
]
response = await vllm_client.chat(messages)
# Track usage for billing
await usage_tracker.log(tenant.id, response.usage)
return responseThe management dashboard lets you onboard new tenants, upload their knowledge base documents, configure system prompts, set rate limits, and view usage analytics. Follow our chatbot server guide for the base chat implementation patterns.
Performance at Scale
On an RTX 6000 Pro 96 GB running Llama 3 8B with vLLM continuous batching, the platform handles 200 concurrent chat sessions across all tenants with a median response latency of 1.4 seconds. Token throughput reaches 4,000 tokens per second shared across active requests. Tenant isolation adds negligible overhead since the only per-tenant work is a vector database query averaging 15 milliseconds.
Usage-based billing tracks tokens consumed per tenant per month. Most operators price at a markup over their infrastructure cost, achieving 60-80% gross margins. The shared model approach means adding a new tenant costs virtually nothing until aggregate concurrency approaches GPU saturation, at which point you add a second GPU node with AI chatbot hosting.
Launch Your Platform
A multi-tenant chatbot platform on dedicated GPU hardware turns a single server into a revenue-generating SaaS product. You control the pricing model, data residency, feature set, and client experience. No per-seat fees from upstream providers eat into your margins. Get started with GigaGPU dedicated GPU hosting and deploy your first tenants this week. Check the vLLM production guide for scaling tips and explore more build patterns in our use case library.
