# Stability Troubleshooting: Models Getting Stuck

## Symptoms
- Model accepts prompt, processes, then sits at 0 t/s indefinitely
- Generation cuts off mid-sentence with no error
- WUI tab hangs forever (no timeout configured)
- llama-server process stays alive but does nothing

## Root Causes (RTX 2080 Ti 11GB)

### 1. MTP Self-Speculation OOM (most common on 35B MoE) — OR Turing Turbo Cache Hang

Two distinct root causes produce the same symptom (0 t/s after prompt processing):

**A) MTP Self-Speculation OOM** — MTP speculative decoding requires additional GPU buffers. On 11GB VRAM, when KV cache grows during long conversations, the MTP head OOMs silently.

**B) Turing Turbo Cache Hang (MORE LIKELY on SM75)** — `--cache-type-k turbo3 --cache-type-v turbo4` + `--spec-type mtp` + `--flash-attn on` triggers an unscheduled shared-memory layout on Turing. The draft head kernel hangs silently. This is NOT VRAM-related — occurs at any context size. MTP overhead is only ~30-60MB (see `mtp-research-turing-2025-06-04.md`). Tracked in llama.cpp issues #9364, #9866, #10024.

**Diagnosis:**
- Run the model at target context without MTP (`--spec-type none`). If it completes reliably, the issue is MTP-related (either OOM or Turbo hang).
- If it still hangs without MTP, check GPU thermal/throttling or context overflow.
- To distinguish OOM vs Turbo hang: check VRAM with `nvidia-smi` during the hang. If VRAM < 10.5GB, it might be OOM. If VRAM is 9-10GB (plenty of headroom), it's the Turbo cache bug.

**Fix (without disabling MTP or reducing context):**\n1. Swap `--cache-type-k turbo3 --cache-type-v turbo4` to `--cache-type-k q8_0 --cache-type-v q8_0` (PRIMARY fix — bypasses the Turing bug entirely, ~0-3% speed loss)\n2. Add `--spec-draft-p-min 0.9` (forces early acceptance, zero speed cost)\n3. Add `--no-cont-batching` (simpler execution flow, no speed impact for single-user)\n4. Reduce `--spec-draft-n-max 2` (lowers draft batch complexity, costs ~10 t/s)\n5. Reduce `--batch-size 512 --ubatch-size 512` (avoids overlapping kernel launches, costs ~12 t/s on 35B MoE)\n\nDo NOT disable flash-attn to fix this — that increases VRAM usage and can cause actual OOM.

**Last resort:** Add `--spec-type none` (costs ~12 t/s on 35B MoE).

### 2. Context Size Exceeds VRAM (35B models at >8K context)
`--ctx-size 250000` on a 35B Q4_K_M with partial offloading pushes KV cache past 11GB during heavy prefills.

**Symptoms:** Model loads fine, generates briefly, then stalls after a few hundred tokens (KV cache fills up during generation).

**Fix:** Reduce `--ctx-size 8192` for 35B models on 11GB. Use 16384 at most. The `--flash-attn on + --cache-type-k turbo3 --cache-type-v turbo4` combo helps but doesn't eliminate the ceiling.

### 3. CPU Thread Contention (partial offloading)
`--threads 16 --threads-batch 32` with partial GPU offloading creates memory bandwidth contention between CPU compute threads and GPU transfer threads.

**Symptoms:** Generation starts fast, gradually slows to 0 t/s over 30-60 seconds.

**Fix:** Reduce to `--threads 6 --threads-batch 6` for all models doing partial GPU offloading. Full-GPU models (gemma-12b, all 9B dense) can keep higher thread counts.
### 4. llama-swap Systemd Watchdog Restart Loop

When a model health watchdog (e.g., Hermes cron `no_agent` script) calls `sudo systemctl restart llama-swap` every N minutes, and the llama-swap systemd service was disabled or in a failed state, two problems arise:

**A) Silent model process kills** — Each restart kills the running llama-swap process (and its child model processes) without warning. The user sees model freezes at random intervals corresponding to the watchdog schedule.

**B) Systemd restart loop** — If a standalone llama-swap holds port 9292, systemd's `Restart=on-failure` spawns a new instance every `RestartSec` seconds that fails with "bind: address already in use". The watchdog sees this as "unhealthy" because `systemctl is-active` returns "inactive" or "activating", triggering yet another restart. Counter can reach 190+ failures.

**Symptoms:**
- Watchdog log shows "RESTARTED: healthy after restart" repeatedly every N minutes
- `journalctl -u llama-swap` shows hundreds of "restart counter" entries with "bind: address already in use"
- Model inference freezes at regular intervals
- `ps aux | grep llama-swap` shows one PID while `systemctl status llama-swap` shows inactive

**Fix:**
```bash
# Kill the standalone process
kill -TERM <standalone-pid>
sleep 3
# Kill any orphaned model processes
pkill -f "llama-server-sm75" 2>/dev/null
# Reset systemd's failed state
sudo systemctl reset-failed llama-swap
# Re-enable and start via systemd
sudo systemctl enable llama-swap
sudo systemctl start llama-swap
# Verify
systemctl is-active llama-swap  # should say "active"
curl -s http://127.0.0.1:9292/v1/models  # should list models
```

After fix, the watchdog checks `systemctl is-active` → returns "active" → exits silently. No more false restarts.

### 5. No End-to-End Timeout (WUI-side)
Open WebUI defaults `AIOHTTP_CLIENT_TIMEOUT=None` (unlimited). Inside the container, the aiohttp client waits forever for the upstream. A stuck llama-server process means the WUI tab hangs permanently.

**Fix:** Set `AIOHTTP_CLIENT_TIMEOUT=300` in the WUI compose env. This ensures stalled generations error out after 5 minutes instead of hanging forever.

### 6. GPU Thermal Throttling
RTX 2080 Ti at sustained load hits 83°C and throttles. Generation t/s approaches zero.

**Symptom:** `nvidia-smi` shows `pstate = P0` but GPU clock drops. Temperature >82°C.

**Fix:** Clean GPU heatsink, improve case airflow, or underclock (e.g., `nvidia-smi -pl 250` to reduce power limit).

## Diagnosis Checklist

```bash
# 1. Is the model process alive?
ps aux | grep [l]lama-server

# 2. What VRAM is in use?
nvidia-smi --query-gpu=memory.used,memory.total,temperature.gpu,pstate --format=csv,noheader

# 3. What flags is the model running with?
ps aux | grep [l]lama-server | head -1 | tr ' ' '\n'

# 4. Is the model actually responding?
curl -s http://127.0.0.1:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<model-id>","messages":[{"role":"user","content":"Reply with exactly OK"}],"max_tokens":5,"stream":false}'

# 5. Is WUI timing out?
docker logs open-webui --tail 10 | grep -i "timeout\|error\|stuck"

# 6. Check llama-swap running model
curl -s http://127.0.0.1:9292/running

# 7. Is systemd in a restart loop?
journalctl -u llama-swap --no-pager -n 10 | grep "Fatal"
```