Small LLM Local Edge Deployment

For edge AI deployments — kiosks, embedded devices, mobile workstations, factory-floor inspection — small LLMs (1B-4B params) provide useful capability without datacenter-tier GPU. Modern small models (Phi-3 Mini, Llama 3.2 1B/3B, Qwen 2.5 0.5B/1.5B/3B) are surprisingly capable.

Models

• Llama 3.2 1B: ~2.5 GB FP16, ~1.3 GB FP8. Fits anywhere.
• Llama 3.2 3B: ~6 GB FP16, ~3 GB FP8.
• Qwen 2.5 0.5B / 1.5B / 3B: similar VRAM tiers; multilingual
• Phi-3 Mini 3.8B: ~7.5 GB FP16, ~3.8 GB FP8. Strong general reasoning at this size.
• Phi-3.5-mini: improved over Phi-3 Mini; same VRAM

Hardware

• RTX 3050 8GB (£79/mo cloud): Phi-3 Mini at FP8 fits comfortably; Llama 3.2 3B fits
• RTX 4060 16GB: 1B-4B models with comfortable headroom
• NVIDIA Jetson: edge-deployment-ready; 1B-3B models run at usable speed
• Mobile workstation NPUs: Apple Silicon M-series, Intel Core Ultra: 1B-3B models with llama.cpp / Ollama
• Raspberry Pi 5: with USB Coral or similar; 0.5B-1B models barely

Workloads

• Structured extraction: 1B-class models with guided decoding handle most extraction reliably
• Classification: works well at 1B-3B with prompt engineering or fine-tune
• Simple Q&A: 3B-4B for KB-grounded Q&A
• Edge transcription post-processing: pair with Whisper Small/Tiny for full edge voice agent
• Don't use for: complex reasoning, long-form writing, code generation (use 7B+ minimum)

Verdict

For edge AI deployments, small LLMs (1B-4B) cover useful workloads at low VRAM and cost. Phi-3 Mini and Llama 3.2 3B are the practical defaults; Qwen 2.5 small variants for multilingual edge. For anything requiring complex reasoning, 7B+ on a proper GPU is necessary.