Linux Kernel Params for GPU

Default Kernel Settings Cost You GPU Performance

Linux ships with kernel defaults optimised for general-purpose computing. On a dedicated GPU server running AI workloads, these defaults leave performance on the table. Small page sizes force excessive TLB misses during large tensor operations. Conservative memory overcommit blocks model loading. Default I/O schedulers add latency to checkpoint writes. Tuning a handful of kernel parameters recovers this lost throughput.

Huge Pages for Large Tensor Allocations

Standard 4 KB pages mean a 32 GB tensor requires 8 million page table entries. Huge pages reduce TLB pressure dramatically:

# Check current huge page config
cat /proc/meminfo | grep Huge
# HugePages_Total:       0
# HugePages_Free:        0
# Hugepagesize:       2048 kB

# Allocate huge pages (2 MB pages)
# For a 64 GB server, reserve 16 GB for huge pages
sudo sysctl vm.nr_hugepages=8192  # 8192 x 2 MB = 16 GB

# Make permanent
echo 'vm.nr_hugepages=8192' | sudo tee -a /etc/sysctl.conf

# For PyTorch — enable transparent huge pages (THP)
# THP automatically promotes regular pages to huge pages
echo 'always' | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo 'defer+madvise' | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

# Make THP settings persistent
cat >> /etc/rc.local <<'EOF'
echo always > /sys/kernel/mm/transparent_hugepage/enabled
echo defer+madvise > /sys/kernel/mm/transparent_hugepage/defrag
EOF

# Verify huge page usage after running model
grep HugePages /proc/meminfo

Memory Overcommit and OOM Tuning

Loading large models temporarily needs more virtual memory than physically available:

# Default overcommit mode (0) is too conservative for AI
# Mode 0: heuristic — kernel guesses if there is enough memory
# Mode 1: always allow — never deny memory requests
# Mode 2: strict — limit to swap + ratio*RAM

# For AI servers: mode 1 prevents spurious allocation failures
sudo sysctl vm.overcommit_memory=1

# Protect the inference process from OOM killer
# Find your inference server PID
VLLM_PID=$(pgrep -f "vllm.entrypoints")
echo -1000 | sudo tee /proc/$VLLM_PID/oom_score_adj

# Lower priority for non-essential processes
echo 500 | sudo tee /proc/$(pgrep prometheus)/oom_score_adj

# Persist via sysctl
cat <<'EOF' | sudo tee -a /etc/sysctl.conf
vm.overcommit_memory=1
vm.overcommit_ratio=95
vm.panic_on_oom=0
vm.oom_kill_allocating_task=1
EOF
sudo sysctl -p

IOMMU and PCIe Settings

GPU DMA transfers benefit from correct IOMMU configuration:

# Check current IOMMU status
dmesg | grep -i iommu

# For best GPU performance, configure IOMMU passthrough
# Edit GRUB boot parameters
sudo sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="[^"]*"/GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on pcie_aspm=off"/' /etc/default/grub

# iommu=pt:        Passthrough mode — DMA goes direct, no translation overhead
# intel_iommu=on:  Enable IOMMU (required for passthrough)
# pcie_aspm=off:   Disable PCIe power saving — prevents latency spikes

# For AMD systems:
# GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt amd_iommu=on pcie_aspm=off"

# Apply GRUB changes
sudo update-grub
sudo reboot

# After reboot, verify
dmesg | grep -i "iommu.*passthrough"

# Set PCIe max read request size for GPU
# Larger reads = better GPU DMA throughput
sudo setpci -s $(lspci | grep NVIDIA | head -1 | awk '{print $1}') 68.w=5000

Scheduler and Network Tuning

Reduce scheduling jitter and network overhead for distributed training:

# I/O scheduler: use none/noop for NVMe (no reordering needed)
echo 'none' | sudo tee /sys/block/nvme0n1/queue/scheduler

# CPU frequency governor: performance mode (no frequency scaling)
echo 'performance' | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Network tuning for multi-GPU communication
cat <<'EOF' | sudo tee -a /etc/sysctl.conf
# Increase network buffer sizes for NCCL
net.core.rmem_max=67108864
net.core.wmem_max=67108864
net.core.rmem_default=33554432
net.core.wmem_default=33554432
net.ipv4.tcp_rmem=4096 87380 67108864
net.ipv4.tcp_wmem=4096 65536 67108864

# Reduce latency
net.ipv4.tcp_low_latency=1
net.ipv4.tcp_nodelay=1

# File descriptor limits for model serving
fs.file-max=2097152
fs.inotify.max_user_watches=524288
EOF
sudo sysctl -p

Verify Kernel Tuning Impact

# Before/after benchmark
python -c "
import torch, time

# Allocate large tensor (simulates model weights)
start = time.time()
x = torch.randn(10000, 10000, device='cuda')
y = torch.mm(x, x)
torch.cuda.synchronize()
print(f'Large matmul: {(time.time()-start)*1000:.1f}ms')

# Memory allocation pattern
start = time.time()
for _ in range(100):
    t = torch.randn(1000, 1000, device='cuda')
torch.cuda.synchronize()
print(f'100 allocations: {(time.time()-start)*1000:.1f}ms')
"

These kernel parameters extract maximum performance from your GPU server. For PyTorch workloads, see the installation guide. Deploy inference with vLLM for production serving. Monitor the impact with our GPU monitoring guide. Browse infrastructure articles, tutorials, and benchmarks for more optimisation.