# Local Model Fleet Management — Systemd Units

## Legacy Systemd Switch Chain (Deprecated)

Only applies to the old per-model systemd setup. Use llama-swap.

```
User picks /model → gateway writes config.yaml → path watcher detects change → auto-model-switch.py stops old service (local→local), waits 3s for VRAM drain, starts new service, polls /health up to 60s → notification
```

## Key Systemd Gotchas

| Issue | Fix |
|-------|-----|
| Unit file not recognized | `sudo systemctl daemon-reload` |
| Daemon-reload needed after every unit change | Always run daemon-reload after add/remove/modify |
| Orphan services after cloud switch | Check `sudo systemctl list-units \| grep llama-server.*running` |
| Rapid-fire config writes | Wait 3s+ between manual writes |
| llama-swap watchdog restart loop | See below |

## Watchdog Restart Loop Fix

When a model health watchdog calls `sudo systemctl restart llama-swap` but systemd conflicts with a standalone process:

**Symptoms:**
- Watchdog logs "RESTARTED: healthy after restart" every N minutes
- `journalctl -u llama-swap` shows 100+ "restart counter" entries with "bind: address already in use"
- Model inference freezes at regular intervals
- `ps aux | grep [l]lama-swap` shows a PID but `systemctl status llama-swap` shows inactive/activating

**Cause:** A standalone llama-swap holds port 9292. systemd's `Restart=on-failure` spawns a new instance every 5s that fails on port conflict.

**Fix:**
```bash
kill -TERM <standalone-pid> && sleep 3
pkill -f "llama-server" 2>/dev/null
sudo systemctl reset-failed llama-swap
sudo systemctl enable llama-swap
sudo systemctl start llama-swap
systemctl is-active llama-swap  # must say "active"
curl -s http://127.0.0.1:9292/v1/models  # must list models
```

## System Tuning
- THP → madvise
- nofile → 65536
- memlock=infinity (in systemd service)

## Diagnostics: "Local Model Not Connecting"

1. **llama-swap running?** `systemctl status llama-swap.service` — ensure active and port :9292 responsive
2. **Model registered?** `curl -s http://127.0.0.1:9292/v1/models | jq '.data[].id'`
3. **Path valid?** Run `scripts/cross-ref-audit.py`
4. **Chat completions work?** curl with a real message through llama-swap
5. **VRAM available?** `nvidia-smi`
6. **THP correct?** `cat /sys/kernel/mm/transparent_hugepage/enabled` — must show madvise

## Pitfall: auto-model-switch.py Path Mismatch
```
can't open file '.../auto-model-switch.py': [Errno 2] No such file or directory
```
**Fix:** Either copy script or disable path watcher (preferred for llama-swap):
```bash
cp ~/.hermes/skills/*/auto-model-switch.py ~/.hermes/scripts/ 2>/dev/null
# OR
systemctl --user disable hermes-config.path hermes-model-switch.service
systemctl --user stop hermes-config.path hermes-model-switch.service
```

## Pitfall: fallback_providers Mismatch
`fallback_providers` must match the exact provider ID. For custom providers, the ID is `custom:<name>.lower().replace(' ', '-')`.