# Hung llama-server Diagnosis

A **hung llama-server** is a process that is alive (visible in `ps`), consuming significant CPU, and accepting TCP connections — but never responding to inference requests. It looks identical to a working server from the shell, but every API call times out.

## Symptom

- `curl` to `/v1/chat/completions` connects successfully but **times out** (no response after 5-30s)
- Process shows **~97-100% CPU** in `ps` or `top`
- `ss -tlnp` shows the port is **listening**
- Hermes reports **"Connection error after 3 retries"** 
- The model-switch log shows `✅ Started` but `API ready: no` or an empty response check

## Diagnosis

```bash
# 1. Check the process is alive and note CPU usage
ps aux | grep llama-server | grep -v grep

# 2. Check the port is listening
ss -tlnp | grep 808

# 3. Test with a short timeout — a working server responds in <5s
#    A hung server connects with zero bytes back
curl -v -m 5 http://127.0.0.1:<PORT>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-3","messages":[{"role":"user","content":"hi"}],"max_tokens":5}' 2>&1 | tail -5

# If "Operation timed out after N ms with 0 bytes received" → hung

# 4. Check process state — "R (running)" with no I/O activity is suspicious
cat /proc/<PID>/status | grep -E 'State|Name'

# 5. Check how long it's been stuck (>few minutes = definitely hung)
ps -o pid,etimes,%cpu,cmd -p <PID>
```

## Root Causes

| Cause | Signature | Fix |
|-------|-----------|-----|
| **GPU compute stall** — CUDA kernel hangs on a bad memory access or driver timeout | Process R state, high CPU, no progress | Kill + restart |
| **Context overrun** — prompt processing exceeds slot and gets stuck in kv_cache resize | Happens on first large prompt after switch | Kill + restart |
| **TurboQuant KV edge case** — turbo4 quantization on a model that doesn't support it cleanly | Only on BoFan fork, certain archs | Switch to `-ctk f16` or different turbo variant |
| **Gemma SWA infinite loop** (old Stormrage fork) | Only Stormrage TQ fork, Gemma 3/4 | Use BoFan MTP fork instead |
| **OOM-triggered wedge** — system runs out of VRAM mid-inference, driver wedges the process | nvidia-smi shows ~99% VRAM usage, process won't die with SIGTERM | `kill -9` required |

## Fix

The hung process typically **won't respond to SIGTERM** (signal 15) because it's in a tight compute loop. Use SIGKILL:

```bash
sudo kill -9 <PID>
```

**Systemd auto-restart:** If the service uses `Restart=on-failure` or `Restart=always` (as most systemd llama-server units do), systemd automatically starts a fresh process within 1-2 seconds. Verify:

```bash
systemctl status <service-name>.service   # Should show "active (running)" with new PID
sleep 5                                    # Wait for model load
curl -m 5 http://127.0.0.1:<PORT>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"hi"}],"max_tokens":5}'
```

## Prevention

- **Add health endpoint timeout** to your monitoring script — check response time, not just TCP connect
- **Cron watchdog** should detect hung state (connect succeeds but no data after N seconds) and restart
- **Avoid context sizes that push VRAM past 95%** — leave headroom for temporary allocations
- **If recurrent with a specific model:** try different KV cache types (`f16` instead of `turbo4`), lower context size, or fewer GPU layers

## Relationship to Hermes Model Switcher

When the model-switcher detects this hung state during a switch:
1. It tries to verify the API endpoint (POST to `/v1/chat/completions`)
2. The request hangs → times out after the script's wait period (~60s)
3. The script logs `API ready: no` and marks the switch as complete anyway
4. Hermes tries to use the endpoint → 3 retries → "Connection error"

**First thing to check** when the user says "Connection error on local model switch": run a direct curl probe (not through Hermes) against the target port. If curl connects but times out, you have a hung server — not a network/config issue.