---
name: llama-swap
description: "Auto-swapping LLM proxy — manages multiple llama.cpp models on debian-ai (:9292), systemd-managed. Replaces individual systemd llama-server services."
version: 1.0.0
author: Hermes Agent
metadata:
  hermes:
    tags: [llama.cpp, proxy, model-swapping, inference]
    related_skills: [llama-cpp, hermes-agent]
---

# llama-swap

Go proxy that auto-swaps llama.cpp models on request — stops current, starts the right one, forwards response.

## Architecture
`Hermes → llama-swap :9292 (systemd) → auto-starts llama-server → proxies`
- `llama-swap` (v226) at `/usr/local/bin/llama-swap`
- Config: `~/.config/llama-swap/config.yaml`
- 7 models managed, no per-model services

## Service Management

**IMPORTANT:** Service has `Restart=always` with 3s delay. `kill` or `pkill` will NOT stop it — systemd respawns immediately. Always use systemctl for lifecycle.

```bash
sudo systemctl stop llama-swap.service    # actually stops it
sudo systemctl restart llama-swap.service # clean restart, picks up config changes
sudo systemctl status llama-swap.service
curl -s :9292/v1/models | jq '.data[].id'
curl -s :9292/running            # active model + args
curl -X POST :9292/api/models/unload

# Upgrade (binary swap)
sudo systemctl stop llama-swap.service
sudo cp /usr/local/bin/llama-swap /usr/local/bin/llama-swap-v<OLD>
- `llama-swap` (v226) at `/usr/local/bin/llama-swap`
- Config: `~/.config/llama-swap/config.yaml`
- **7 models** managed (as of June 21): qwen36-35b-mtp, gemma-12b, gemma-26b-200k, nemotron-term-14b, nemotron-3-nano-30b, qwen3-coder-next, glm-4.7-flash-reap
- All profiles run on **one unified binary**: `llama-server-sm75` (b9743, built from upstream master, sm75-optimized)
- Old split (sm75 b9341 fork + upstream ac4cdde) retired

## Model Config Format
```yaml
qwen36-35b-mtp:
  cmd: /usr/local/bin/llama-server-sm75 -m /models/...gguf --port ${PORT} --host 127.0.0.1 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 --no-mmap --mlock --jinja --reasoning off --prio 2 --poll 30 --no-cont-batching --timeout 300 -c 250000 --threads 16 --threads-batch 32 --n-cpu-moe 128 --batch-size 2048 --ubatch-size 512 --temp 0.2 --top-p 0.95 --min-p 0.05 --top-k 20 --metrics --no-host -fitt 512 --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-n-min 0.75
  aliases: [qwen36, qwen36-mtp]
  ttl: 3600
```

## Config Traps

- **`sendLoadingState: false`** — Default is `true`. When true, llama-swap sends SSE loading-state events (`{"status":"loading","model":"X"}`) during model swap. Open WebUI renders this as a loading indicator. **This is NOT where loading-screen jokes come from** — those come from the model's own thinking/reasoning output (see "Model Thinking Behavior" below). Set to `false` if the loading indicator is distracting in the UI.
  
  **CRASH + sendLoadingState interaction**: When a model server crashes during load (`exited prematurely`), llama-swap sends a loading-state SSE but never sends a completion. Open WebUI renders the loading-state text (e.g. "llama-swap loading model: X") as if it were the model's output, wrapped in a `<details type="reasoning">` block. To the user it looks like the model responded with loading-screen text. Always check `journalctl -u llama-swap` for `exited prematurely` when a model produces no real response.
  
- **Qwen models always emit thinking tags** — `--reasoning off` in llama.cpp controls the server's REASONING FEATURE (token-level), NOT the model's native output of `<think>`/`<thinking>` tags. Qwen models (qwen36, qwen3-coder-next, etc.) are fine-tuned to emit these tags regardless. Open WebUI's middleware then renders them, creating the appearance of "thinking" even with `--reasoning off`. The fix is on the client side (disable Thinking toggle or filter tags upstream).

- **GLM-4.7-flash-reap: `--reasoning off` is REQUIRED** — GLM natively outputs content via the `reasoning_content` JSON field instead of `content`. Without `--reasoning off`, the `content` field is empty and `reasoning_content` contains the model's chain-of-thought. Adding `--reasoning off` forces all output into the `content` field.

- **GLM-4.7-flash-reap OOM: q8_0 KV cache at 131K ctx = ~3.6 GB** — On 11 GB RTX 2080 Ti, this OOMs because model weights (~9-10 GB) plus q8_0 KV cache (~3.6 GB) exceed available VRAM. Use `--cache-type-k q4_0 --cache-type-v q4_0` instead to halve KV cache to ~1.8 GB. (See `local-model-fleet-management` for the general OOM escalation path.)
- `healthCheckTimeout: 120`, `performance.every: 15s`
- Single-line `cmd:` — `env` in cmd doesn't work.

## Cross-Config Sync
After any config change, verify all 3 sources match:
```bash
grep -E '\-c |--ctx-size' ~/.config/llama-swap/config.yaml
curl -s :9292/running | python3 -c "import sys,json; [print(r['cmd']) for r in json.load(sys.stdin).get('running',[])]"
grep -A1 context_length ~/.hermes/config.yaml
```

## Reference Files
- `references/llama-swap-config.md` — Options, canonical configs, model table, frontend integration
- `references/model-thinking-behavior.md` — Which models emit native thinking tags, how Open WebUI renders them, loading-screen feedback loop
- `references/operations.md` — Upgrade, verification, migration, port allocation
- `references/troubleshooting.md` — Config drift, cold load, session bloat, OOM, MTP bugs, UFW
- `references/ufw-docker.md`, `references/vram-breakdown-dense-14b.md`, `references/fleet-benchmarks-2080ti-may2026.md`, `references/bonsai-prismml-setup.md`, `references/2026-06-19-fleet-audit-devstral-moonlight.md`, `references/fleet-model-specs-2026-06-27.md`
- `scripts/benchmark_models.py`