# System-Level Tuning for LLM Inference

## Transparent Hugepages (THP)

**Default `always` hurts llama.cpp.** `--mlock` + THP `always` causes memory fragmentation and inflated RSS. Set to `madvise`:

```bash
# Immediate (non-persistent):
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

# Persistent (systemd oneshot):
cat > /etc/systemd/system/thp-fix.service << 'EOF'
[Unit]
Description=Set transparent hugepages to madvise for ML workloads
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled && echo madvise > /sys/kernel/mm/transparent_hugepage/defrag'
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable thp-fix.service --now
```

**Verification:** `cat /sys/kernel/mm/transparent_hugepage/enabled` → should show `always [madvise] never`

**Note:** Kernel 6.12+ uses `/sys/kernel/mm/transparent_hugepage/`. Older kernels may use `/sys/kernel/mm/transparent_hugepage/enabled`. Sysctl paths (`vm.transparent_hugepage_enabled`) don't work on modern kernels.

## ulimits in systemd

llama.cpp with `--cont-batching` needs many file descriptors. Default `LimitNOFILE=1024` is too low:

```
[Service]
LimitNOFILE=65536
LimitMEMLOCK=infinity
```

`memlock=infinity` is required because `--mlock` locks model weights in RAM.

## CPU Governor

`schedutil` is the best default for mixed workloads. `performance` can help raw throughput on dedicated inference boxes but wastes power. Check:

```bash
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
```

## NUMA

Single-socket systems: no NUMA tuning needed. Multi-socket: pin llama-server to the NUMA node closest to the GPU via `numactl --cpunodebind=0 --membind=0`.

## Quick Health Check

```bash
# THP
cat /sys/kernel/mm/transparent_hugepage/enabled

# ulimits (for running process)
cat /proc/$(pgrep llama-swap)/limits | grep -E "open files|locked memory"

# Governor
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# GPU persistence
nvidia-smi -q | grep "Persistence Mode"
```