Gradio AI Demo: Deployment on GPU

You will build a Gradio demo for a GPU-hosted model and deploy it on a dedicated server with proper queuing, authentication, and reverse proxy configuration. By the end, you will have a production-ready demo accessible over HTTPS that handles concurrent users without crashing.

Building a Chat Interface

Gradio’s ChatInterface component creates a full-featured chat UI with message history, streaming, and retry buttons in a few lines of code.

import gradio as gr
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

def chat(message, history):
    messages = [{"role": "system", "content": "You are a helpful assistant."}]
    for user_msg, bot_msg in history:
        messages.append({"role": "user", "content": user_msg})
        messages.append({"role": "assistant", "content": bot_msg})
    messages.append({"role": "user", "content": message})

    stream = client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=messages,
        stream=True, max_tokens=1024
    )
    partial = ""
    for chunk in stream:
        if chunk.choices[0].delta.content:
            partial += chunk.choices[0].delta.content
            yield partial

demo = gr.ChatInterface(
    fn=chat,
    title="LLaMA 3.1 Chat",
    description="Self-hosted LLM inference on dedicated GPU"
)
demo.launch(server_name="0.0.0.0", server_port=7860)

This connects to a vLLM backend through the OpenAI-compatible API. For backend setup, follow the vLLM production guide.

Advanced Layouts with Blocks

The Blocks API gives full control over layout, allowing multi-model demos, side-by-side comparisons, and tabbed interfaces.

with gr.Blocks(title="AI Demo Hub") as demo:
    gr.Markdown("# AI Model Demo Hub")

    with gr.Tab("Chat"):
        chatbot = gr.Chatbot(height=400)
        msg = gr.Textbox(placeholder="Type your message...")
        clear = gr.ClearButton([msg, chatbot])

        def respond(message, chat_history):
            response = client.chat.completions.create(
                model="meta-llama/Llama-3.1-8B-Instruct",
                messages=[{"role": "user", "content": message}],
                max_tokens=512
            )
            bot_reply = response.choices[0].message.content
            chat_history.append((message, bot_reply))
            return "", chat_history

        msg.submit(respond, [msg, chatbot], [msg, chatbot])

    with gr.Tab("Summarise"):
        input_text = gr.Textbox(lines=10, label="Input Text")
        output_text = gr.Textbox(lines=5, label="Summary")
        summarise_btn = gr.Button("Summarise")

        def summarise(text):
            response = client.chat.completions.create(
                model="meta-llama/Llama-3.1-8B-Instruct",
                messages=[{"role": "user", "content": f"Summarise this:\n{text}"}],
                max_tokens=256
            )
            return response.choices[0].message.content

        summarise_btn.click(summarise, input_text, output_text)

Queue and Concurrency

Gradio’s queue system manages concurrent users. Without it, simultaneous requests can overwhelm the GPU. Configure the queue based on your hardware capacity.

demo.queue(
    max_size=20,           # Maximum queued requests
    default_concurrency_limit=1  # Process one GPU request at a time
)
demo.launch(
    server_name="0.0.0.0",
    server_port=7860,
    max_threads=10,        # HTTP handling threads
    show_error=True
)

Set default_concurrency_limit=1 for GPU-bound tasks to prevent VRAM exhaustion from parallel inference. For CPU-bound preprocessing, increase it. For async queue patterns at larger scale, see the Redis queue guide.

Authentication

Protect your demo with built-in authentication or integrate with your existing auth system.

# Simple username/password auth
demo.launch(
    auth=("admin", "your-secure-password"),
    auth_message="Enter credentials to access the AI demo"
)

# Multiple users
demo.launch(
    auth=[("user1", "pass1"), ("user2", "pass2")]
)

# Custom auth function
def auth_check(username, password):
    # Verify against your database or LDAP
    return username == "admin" and password == "secure"

demo.launch(auth=auth_check)

Nginx and HTTPS Deployment

Place Gradio behind Nginx for TLS termination, WebSocket proxying, and static asset caching.

server {
    listen 443 ssl;
    server_name demo.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/demo.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/demo.yourdomain.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:7860;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 300s;
    }
}

The WebSocket upgrade headers are critical — Gradio uses WebSockets for streaming and queue updates. Without them, the interface loads but inference hangs.

Production Tips

Run Gradio as a systemd service for automatic restarts. For comparing Gradio with Streamlit, see the framework comparison. Add GPU monitoring to track inference latency per user. The self-hosting guide covers server configuration, and our tutorials section has additional deployment patterns. For a complete API alongside your demo, add a FastAPI inference server.