What You’ll Connect
After this guide, your PagerDuty instance will receive and route alerts from your GPU AI infrastructure — so the right engineer gets paged when inference goes down, GPU memory fills up, or hardware errors occur. Your dedicated GPU servers running vLLM or Ollama become fully integrated into your incident management workflow.
The integration sends alerts to PagerDuty’s Events API from a monitoring script running on your GPU server. PagerDuty handles escalation policies, on-call scheduling, and incident tracking — you focus on defining what constitutes a GPU infrastructure emergency.
Health Check Script –> PagerDuty Events API –> Incident Created | (cron every 60s) POST /v2/enqueue | GPU metrics checks GPU health, On-call engineer vLLM status inference availability gets paged via disk space sends alert if down phone/SMS/app | | Auto-resolve <-- Script detects <-- Recovery check <-- Issue resolved in PagerDuty service restored passes -->Prerequisites
- A GigaGPU server running an inference endpoint (vLLM production guide)
- A PagerDuty account with at least one service configured
- A PagerDuty integration key (Events API v2 type) for your service
- Python 3 or bash scripting capability on the GPU server
- GPU monitoring tools:
nvidia-smiavailable (standard with NVIDIA drivers)
Integration Steps
Create a new PagerDuty service for your GPU AI infrastructure (or use an existing one). Under Integrations, add an “Events API v2” integration and copy the routing key. This key authenticates alert submissions from your GPU server.
Write a health check script that runs on the GPU server via cron. The script checks three conditions: GPU hardware health via nvidia-smi, inference endpoint availability via an HTTP request, and disk space for model storage. If any check fails, the script sends a trigger event to PagerDuty. When all checks pass after a previous failure, it sends a resolve event.
Configure PagerDuty’s escalation policy to page the appropriate on-call engineer. Set urgency rules so GPU hardware failures page immediately (high urgency) while latency degradation creates a low-urgency notification for review during business hours.
Code Example
Health check script that monitors your AI inference server and alerts PagerDuty:
#!/usr/bin/env python3
"""GPU AI infrastructure health check for PagerDuty alerting."""
import subprocess, requests, json, os
PD_ROUTING_KEY = os.environ["PAGERDUTY_ROUTING_KEY"]
GPU_API_URL = "http://localhost:8000/v1/models"
GPU_API_KEY = os.environ.get("GPU_API_KEY", "")
STATE_FILE = "/tmp/pd_gpu_alert_state"
def check_gpu():
result = subprocess.run(["nvidia-smi", "--query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu",
"--format=csv,noheader,nounits"], capture_output=True, text=True)
if result.returncode != 0:
return False, "nvidia-smi failed — GPU hardware error"
lines = result.stdout.strip().split("\n")
for line in lines:
util, mem_used, mem_total, temp = [float(x.strip()) for x in line.split(",")]
if temp > 90:
return False, f"GPU temperature critical: {temp}C"
if mem_used / mem_total > 0.98:
return False, f"GPU memory near full: {mem_used}/{mem_total} MB"
return True, "GPU healthy"
def check_inference():
try:
headers = {"Authorization": f"Bearer {GPU_API_KEY}"} if GPU_API_KEY else {}
resp = requests.get(GPU_API_URL, headers=headers, timeout=10)
return resp.status_code == 200, f"Inference endpoint status: {resp.status_code}"
except Exception as e:
return False, f"Inference endpoint unreachable: {e}"
def send_pagerduty(action, summary, severity="critical"):
requests.post("https://events.pagerduty.com/v2/enqueue", json={
"routing_key": PD_ROUTING_KEY,
"event_action": action,
"dedup_key": "gpu-inference-health",
"payload": {
"summary": summary,
"severity": severity,
"source": "gpu-inference-server",
"component": "vllm",
}
})
gpu_ok, gpu_msg = check_gpu()
inf_ok, inf_msg = check_inference()
all_ok = gpu_ok and inf_ok
if not all_ok:
send_pagerduty("trigger", f"{gpu_msg} | {inf_msg}")
open(STATE_FILE, "w").write("alert")
elif os.path.exists(STATE_FILE):
send_pagerduty("resolve", "GPU and inference restored")
os.remove(STATE_FILE)Testing Your Integration
Run the script manually while your inference server is healthy — no alert should fire. Stop the vLLM container and run the script again — a PagerDuty incident should appear within seconds. Restart vLLM and run the script once more to verify the auto-resolve works and the incident closes in PagerDuty.
Add the script to cron (* * * * * for every-minute checks) and test by temporarily blocking access to the inference port. Confirm the PagerDuty notification reaches your on-call channel (phone, SMS, or app push).
Production Tips
Use PagerDuty’s dedup_key feature (already included in the script) to prevent alert storms. Multiple failures for the same issue create a single incident rather than flooding the on-call engineer with duplicate pages. The key groups related failures together.
Create separate PagerDuty services for different severity levels: GPU hardware failures go to a high-urgency service that pages immediately, while performance degradation goes to a lower-urgency service that creates tickets for business-hours review.
For teams operating multiple open-source model servers, extend the script to check each server and include the hostname in the dedup_key. This creates separate incidents per server while maintaining the same escalation workflow. Secure your endpoints with our API guide, explore more tutorials, or get started with GigaGPU to build resilient GPU AI infrastructure.
