Text Generation WebUI as a Production API

The “text-generation-webui” project (often “oobabooga”) is built around a UI for experimentation. The same codebase exposes an OpenAI-compatible API that runs fine in production on dedicated GPU hosting. It is not the first recommendation for heavy traffic but it has genuine strengths for certain deployments.

Contents

• When it is the right tool
• Production setup
• vs vLLM / TGI

When It Fits

text-generation-webui is good when:

• You want both a UI (for internal ops) and an API (for integrations) from one process
• You run GGUF, GPTQ, AWQ, ExLlamaV2, and Transformers loaders interchangeably
• You swap models frequently and want hot-reload via the UI
• Traffic is modest (under 20 concurrent users)

It is not the right tool when you need the top-tier throughput vLLM or TGI deliver, or when you are serving hundreds of concurrent users.

Production Setup

python server.py \
  --model Meta-Llama-3.1-8B-Instruct-exl2 \
  --loader exllamav2 \
  --listen --listen-host 0.0.0.0 \
  --api --api-port 5000 \
  --api-key your-secret-key \
  --gpu-memory 20 \
  --no-stream

Key flags:

• --api enables the OpenAI-compatible endpoint
• --api-key adds bearer-token auth
• --listen-host 0.0.0.0 binds externally
• --gpu-memory caps VRAM per GPU in GB

Put nginx or Caddy in front for TLS and rate limiting. See vLLM behind nginx – the pattern is the same.

Versus vLLM / TGI

See OpenAI-compatible API guide and vLLM args reference.