Pen Testing Your AI Inference Stack

Your annual penetration test covers the web application and database. The testers never touch the vLLM endpoint running on port 8000. The API gateway forwards requests to the inference server without input validation. A prompt injection bypasses your system prompt and extracts the complete system instructions, including internal API endpoints mentioned in the prompt. Your AI inference stack has an attack surface that traditional pen tests miss entirely. This guide covers how to test it on self-hosted GPU infrastructure.

AI Attack Surface Mapping

Before testing, map the complete attack surface of your AI deployment. It extends well beyond the inference API:

Engage a penetration testing firm with AI/ML expertise, or train your internal team on AI-specific attack techniques. Standard web application testers will miss model-layer vulnerabilities.

Model-Layer Testing

Model-layer attacks target the AI model itself, not just the infrastructure. Test for prompt injection (can an attacker override system instructions through user input?), jailbreaking (can the model be coerced into producing harmful or policy-violating output?), data extraction (can the model be prompted to reveal training data or system prompts?), and model denial of service (can a crafted input cause excessive generation, consuming all GPU memory?).

Use frameworks like Garak or PyRIT for systematic prompt injection testing. Run thousands of known injection payloads against your vLLM endpoint and categorise the results. This is not a one-time test — injection techniques evolve, so schedule quarterly model-layer assessments. See our prompt injection protection guide for defensive measures.

API Security Testing

The inference API is the primary network attack surface. Test authentication bypass (can unauthenticated requests reach the model?), authorisation flaws (can one user’s API key access another user’s conversation history?), rate limiting (can an attacker exhaust GPU resources through rapid requests?), input validation (does the API reject oversized prompts, malformed JSON, or unexpected content types?), and response filtering (do responses ever contain internal system information?).

Use Burp Suite or OWASP ZAP with custom rules for AI endpoints. Standard web security scanners will not generate meaningful prompts, so combine automated scanning with manual testing. On private infrastructure, you can test aggressively without affecting other tenants.

Infrastructure Testing

The underlying GPU server is a Linux machine with NVIDIA drivers and CUDA libraries — a substantial attack surface. Test for SSH configuration weaknesses (password auth enabled, weak ciphers), NVIDIA driver vulnerabilities (check against known CVEs), Docker container escapes (if running models in containers), kernel vulnerabilities (privilege escalation from application to root), and network segmentation bypass (can the inference server reach unintended networks?).

Run Nessus or OpenVAS vulnerability scans against the GPU server. Test from both external (internet-facing) and internal (same network) perspectives. The internal test is critical — if an attacker compromises the application server, can they pivot to the GPU server?

Supply Chain Testing

AI systems have a unique supply chain risk: model weights downloaded from public repositories. Test for model integrity verification (are SHA-256 checksums validated at download?), dependency security (are Python packages pinned to exact versions with hash verification?), container image provenance (are Docker images built from verified base images?), and driver authenticity (are NVIDIA drivers downloaded from official sources and verified?).

Test what happens if a model file is corrupted or replaced. Does the system detect the modification? Does it refuse to load a model with a mismatched checksum? These controls prevent both accidental corruption and deliberate supply chain attacks.

Remediation and Retesting

Prioritise findings by exploitability and impact. Critical findings (unauthenticated API access, prompt injection leading to data extraction) require immediate remediation. High findings (missing rate limiting, weak SSH configuration) should be resolved within 30 days. Document all findings, remediation actions, and retest results. This documentation satisfies GDPR Article 32 requirements and supports compliance across sectors. Schedule retesting 30 days after remediation to verify fixes. Teams running customer-facing chatbots or document processing should include these workloads in every penetration test cycle.