---
name: local-model-fleet-management
description: "Manage your local LLM inference fleet — VRAM-conscious service lifecycle (systemd, GGUF alignment, path watchers) combined with smart model selection and routing strategy for cost-effective task dispatch."
version: 1.0.0
tags: [model-management, systemd, llama-server, vram, model-routing, cost-management]
---

# Local Model Fleet Management

## When to Use

- Setting up or modifying the local inference stack (llama-swap, systemd, model paths)
- Diagnosing model switch failures, cold load hangs, or VRAM contention
- Selecting a model for a task: local-first, escalate only when capability or speed demands
- Auditing three-way alignment: disk ↔ llama-swap config ↔ Hermes custom_providers
- GPU migration (e.g., Pascal → Turing) — re-benchmark every model afterward
- Binary update audit — checking whether llama.cpp builds need updating

## Core Workflow Rule: Research Before Building

The user has explicitly stated: **"stop guessing and creating things, always web research when you get stuck."**

This means:
- NEVER assume CLI flag names, build targets, or binary capabilities from memory alone.
- When uncertain about a flag, build target, or architecture support: consult the actual binary's `--help` output first, then web research (GitHub PR, HF model card, Unsloth docs).
- For model setup: read the HF model card README.md via raw URL before proposing flags.
- For build targets: check the GitHub PR's file tree and release notes before building.
- "Web research" means: GitHub issues/PRs, HF model cards, Unsloth docs, release notes — not asking the user to confirm guesses.
- If a build fails (e.g., gcc version mismatch), research the fix (env var, flag, compiler switch) rather than guessing alternatives.

## Full Fleet Stress-Test Pipeline

When the user asks to "stress test", "audit", "bench test", or "scorecard" all models: see `references/fleet-stress-test-methodology.md` for the complete 7-phase pipeline (research → audit → redundancy assessment → benchmark → scorecard → cleanup → tweak audit).

The benchmark script at `scripts/bench-fleet-v2.py` runs a single pass per model through llama-swap: cold load timing, generation speed (500 tok, dual run), prompt processing speed, and VRAM capture. Run it any time the fleet changes or the user asks for fresh numbers:
```bash
python3 -u ~/.hermes/skills/devops/local-model-fleet-management/scripts/bench-fleet-v2.py
```

### Quant Upgrade Trade-off Analysis

When evaluating whether a lower quantization (Q6→Q5, Q8→Q6) can fund a context increase or faster inference:

1. **Get file sizes** for both quants from the HF repo
2. **Calculate VRAM delta:** dense models ∆VRAM ≈ ∆file_size; MoE models with --n-cpu-moe save proportionally less (only non-expert layers plus expert layers moved to GPU)
3. **KV cache budget:** `2 × n_layers × n_kv_heads × head_dim × target_ctx` bytes at q8_0
4. **Ceiling check:** total must be < 10,500 MiB on 11 GB card (including 500 MiB overhead)

**Real example (Ornith-9B, RTX 2080 Ti):** Q6→Q5 freed ~0.5 GB VRAM, enabling 32K→64K context with on-GPU KV cache, eliminating CPU KV bottleneck.

### Per-Model Tweak Audit

After benchmarking, compare each model's current flags against model-card recommendations. Common issues found in the wild:
- **Temperature mismatch** — Gemma 4 recommended temp=1.0, often run at 0.2
- **MTP draft-n-max too high** — n=2 often outperforms n=3-5 on 12GB VRAM
- **`--reasoning off` missing** on GLM models → empty `content` field
- **`--no-kv-offload` left on** after context reduction freed VRAM
- **VRAM margin < 200 MiB** → compute buffer OOM risk, add `--no-warmup`

### Tweak → Re-Benchmark Workflow

After applying changes to model flags (temp, MTP draft length, context size, batch size, etc.), always re-benchmark to measure the impact. Use the same benchmark script for apples-to-apples comparison:

```bash
# Before tweaks
python3 -u ~/.hermes/skills/devops/local-model-fleet-management/scripts/bench-fleet-v2.py

# Apply tweaks, restart llama-swap
...

# After tweaks
python3 -u ~/.hermes/skills/devops/local-model-fleet-management/scripts/bench-fleet-v2.py
```

**What to look for in the before/after comparison:**
- **Gen t/s (cold):** first-request speed after cold load — affected by --no-warmup, batch size
- **Gen2 t/s (warm):** second-request speed — better measure of MTP draft acceptance, KV cache efficiency
- **PP t/s:** prompt processing — affected by context size, n-cpu-moe, batch/ubatch
- **VRAM/Free:** VRAM margin — anything <200 MiB free is a crash risk
- **MTP n=2 vs n=4:** warm gen often improves with n=2; cold gen may dip slightly
- **Temp change:** should not affect speed measurably, but large swings (>0.2 to >1.0) change output token distribution which can affect acceptance rates

**Real example from Jun 30 session (RTX 2080 Ti):**
| Model | Change | Before t/s | After t/s | Δ |
|-------|--------|:----------:|:---------:|:-:|
| Qwythos-9B | MTP n=4→2 | 95.2 | 86.6 | -9% cold |
| Qwythos-9B (warm) | MTP n=4→2 | 81.9 | **91.9** | **+12% warm** |
| Ornith-9B | 64K→96K ctx | 69.2 | **69.7** | **~same** (free context!) |

## Key Commands

- `systemctl status llama-swap.service` — verify the proxy is active
- `curl -s http://127.0.0.1:9292/v1/models | jq '.data[].id'` — list registered models
- `find /models/downloads/ -name '*.gguf' -maxdepth 2 | sort` — models on disk
- `grep -oP -- '-m \S+' ~/.config/llama-swap/config.yaml | while read f p; do [ -f "$p" ] && echo "✓ $p" || echo "✗ MISSING: $p"; done` — path audit
- `scripts/cross-ref-audit.py` — full three-way alignment check
- `nvidia-smi` — check VRAM availability before loading
- `cat /sys/kernel/mm/transparent_hugepage/enabled` — must show `madvise`

## New Model Integration Procedure

When adding a new GGUF model to the fleet, follow this checklist:

### 1. Research Architecture
```bash
# Get the original model config
curl -sL "https://huggingface.co/<org>/<model>/raw/main/config.json" | python3 -c "
import sys,json; d=json.load(sys.stdin)
for k in ['model_type','architectures','num_hidden_layers','num_attention_heads',
          'num_key_value_heads','hidden_size','max_position_embeddings',
          'num_experts','num_experts_per_tok']:
    if k in d: print(f'{k}: {d[k]}')
if 'text_config' in d: tc=d['text_config']; print(f'text_config.model_type: {tc.get(\"model_type\")}'); print(f'text_config.num_experts: {tc.get(\"num_experts\")}'); print(f'KV heads: {tc.get(\"num_key_value_heads\")}')
"
```

### 1b. Read the Model Card
Always read the HF model card README.md via raw URL for recommended flags and notes:
```bash
curl -sL "https://huggingface.co/<org>/<model>/raw/main/README.md" | head -200
```
Look for: recommended context size, sampling params (temp/top-p/top-k), architecture changes (`gemma4_unified` vs `gemma3`), MTP availability, and any special requirements ("needs recent llama.cpp").
### 1c. Parse GGUF Metadata from Raw File

Use the dedicated `scripts/gguf-metadata-scan.py` script to extract architecture, context length, layer count, head counts, quantization, and HF source from GGUF files. It handles GGUF v3 format (u64 key lengths), big tokenizer arrays (100K+ vocab), and partial buffers:

```bash
# Scan all models in the fleet
python3 ~/.hermes/skills/devops/local-model-fleet-management/scripts/gguf-metadata-scan.py

# Output: formatted Markdown table (TTY) or JSON (piped)
python3 scripts/gguf-metadata-scan.py | jq '.[] | {file, architecture, context_length, block_count, quant, disk_gb}'

# Scan specific files
python3 scripts/gguf-metadata-scan.py /models/downloads/Qwen3.6-35B*.gguf
```

Key differences from the `gguf` Python package:
- No Python `gguf` module dependency (which can hang on large files >5 GB)
- Reads only the metadata header (~2 MB), skips tensor data entirely
- Prints formatted table or JSON, suitable for fleet audits and inventory reports

### 2. Check Binary Compatibility
```bash
# Check if the model_type is supported by each binary
strings /usr/local/bin/llama-server-sm75 | grep -c "<model_type>"    # sm75 support count
strings /usr/local/bin/llama-server-upstream | grep -c "<model_type>"  # upstream support count
```
- If sm75 has 0 refs and upstream has 74+ (nemotron_h pattern), it's upstream-only
- If both have refs, either binary works
- For sm75: check if `--spec-type mtp` is needed (check `--help 2>&1 | grep spec-type`)
- For upstream: check if `--spec-type draft-mtp` is needed

### 3. Check MTP Support
- Search the GGUF filename for "MTP" 
- Check Unsloth docs or README for MTP mentions
- Dense Qwen3 models from mradermacher or bartowski typically do NOT have MTP heads
- Unsloth GGUF variants often include MTP

### 4. Get Real File Size
```bash
curl -sIL "https://huggingface.co/<org>/<model>/resolve/main/<file>.gguf" 2>&1 | grep -i x-linked-size
```
Divide by 1024^3 for GB. This gives real file size, not the HF API 0-byte bug.

### 5. Estimate VRAM Budget
**For dense models:** VRAM ≈ file_size * (ngl / total_layers) + KV_cache + 0.5GB overhead
**For MoE models with --n-cpu-moe 128:** VRAM ≈ 3GB (active) + KV_cache + 0.5GB overhead

KV cache per token (q8_0) = 2 * n_layers * n_kv_heads * (hidden_size / n_attention_heads) bytes

**Efficiency heuristic:** Models with very few KV heads (e.g., Nemotron-3-Nano with 2 KV heads) have extremely small KV caches and can handle massive context on limited VRAM. This is the first thing to check.

### 6. Multi-Platform Registration

After the model is verified in llama-swap, register it on all three platforms so it's selectable everywhere:

#### 6a. Hermes custom_providers
```python
import yaml, os
path = os.path.expanduser("~/.hermes/config.yaml")
with open(path) as f:
    data = yaml.safe_load(f)
for cp in data.get("custom_providers", []):
    if cp.get("name") == "Local LLM (llama-swap)":
        cp["models"].append({
            "context_length": <CTX>,
            "description": "<Short description>",
            "id": "<llama-swap-model-key>",
            "name": "<Human-readable name>"
        })
        break
with open(path, "w") as f:
    yaml.dump(data, f, default_flow_style=False, sort_keys=False, width=1000)
# Validate
python3 -c "import yaml; yaml.safe_load(open('/home/rurouni/.hermes/config.yaml'))" && echo "Valid"
```

#### 6b. OpenCode local-gateway
Edit `~/.config/opencode/opencode.json` and add a model entry under `provider.local-gateway.models`:
```json
"<model-id>": {
  "name": "<Human-readable name>",
  "limit": {
    "context": <CTX>,
    "output": 16384
  }
}
```

#### 6c. Open WebUI
Open WebUI points to llama-swap via `OPENAI_API_BASE_URL=http://host.docker.internal:9292/v1` and auto-discovers models from `/v1/models`. Restart the container to refresh the list:
```bash
docker restart open-webui
```
Wait 15s then verify at http://localhost:3081 (model picker should show the new entry).

### 7. Verify the Full Switch Path

### 7. OOM Escalation Path (DO NOT use q4_0 KV first)
If a model OOMs on load:
1. **Reduce context** first (e.g., 256K → 160K → 128K → 64K → 16K)
2. **Raise -fitt** from 512 to 768/1024 (increases safety margin)
3. **Reduce draft aggressiveness** (lower --spec-draft-n-max, raise --spec-draft-p-min) or remove --spec-draft-ngl
4. **Add --no-kv-offload** (moves KV cache to system RAM, slower but works)
5. **Only then** test turbo cache or q4_0 KV cache

**Exception: GLM-4.7-flash-reap.** At 131K context with q8_0 KV cache, GLM needs ~3.6 GB for KV alone. Two valid fix paths:
- **Path A (keep q8_0 quality):** increase `--n-cpu-moe` from 20 to 30 — frees ~0.8 GB by offloading 10 more MoE experts to CPU. Speed drops ~20% (53.5 → 42.7 t/s).
- **Path B (keep speed):** downgrade KV cache to `q4_0/q4_0` — frees ~1.8 GB. Minor quality impact on long-context retrieval. Speed stays at ~53.5 t/s.
**IMPORTANT:** GLM also requires `--reasoning off` — without it, the model only outputs `reasoning_content` (empty `content` field = no visible response).

## Known CUDA Regressions

See `references/cuda-regressions-known.md` for tracked upstream issues affecting RTX 2080 Ti performance: #24514 (25% perf drop b9301→b9305) and #24670 (draft-mtp not activating on Turing sm_75).

## Research Agent Hallucination Warning

Delegate_task subagents for web research **invent flags that don't exist**. Verified in June 2026:
- `--cuda-streams N` — does not exist on either llama.cpp binary
- Always verify proposed flags against actual binary --help output before applying

### Pitfall: `huggingface-cli` Deprecated (June 2026)

`huggingface-cli` is deprecated and no longer works. Use the new `hf` CLI instead:

```bash
# OLD (broken):
huggingface-cli download <org>/<repo> <file> --local-dir ./dir

# NEW:
hf download <org>/<repo> --include "<file>" --local-dir ./dir
```

The `hf` CLI is installed alongside `huggingface_hub`. Use `hf --help` for commands.
Key differences:
- Positional file args work directly: `hf download <org>/<repo> <file> --local-dir ./dir`
- `--local-dir-use-symlinks` does NOT exist on `hf` (omit it)
- Use `--force-download` to redownload cached files
- `--include` supports glob patterns (alternative to positional args)

## Pitfall: --poll and --n-cpu-moe Misunderstandings

From the binary --help output:
- `--poll <0...100>` — polling **level**, not milliseconds. Default 50. 30 recommended for single-user SM75.
- `--n-cpu-moe N` — keep the MoE weights of the **first N layers** in the CPU. NOT CPU thread count. NOT "expert threads".
- Setting n-cpu-moe too high (e.g., 128 on a 40-layer model) pushes all MoE experts to CPU. Safe but slow.
- Setting n-cpu-moe too low risks OOM if the model + KV cache exceed VRAM.

### --n-cpu-moe Performance Tuning

When a model has VRAM headroom, reducing `--n-cpu-moe` below the layer count moves expert weights to GPU, significantly improving speed:

**Methodology:**
1. Note current VRAM usage with `--n-cpu-moe 128` (all experts on CPU)
2. The model's layer count determines the max n-cpu-moe that keeps experts on CPU
   - `--n-cpu-moe N` where N >= layer_count = all experts on CPU
   - `--n-cpu-moe N` where N < layer_count = layers (N..end) experts on GPU
3. Try progressively lower values (e.g., 40 → 35 → 30 → 20) with `-c 16384` for fast testing
4. Check VRAM — if it fits in ~9-10 GB, benchmark to see if speed improved
5. If OOM, increase n-cpu-moe by 2-3

**Diminishing returns:** Models where experts are a small fraction of total parameters (e.g., coder-next-80b with 80B total / 3B active) gain less from moving a few layer experts to GPU because most compute time is still in the CPU-offloaded experts. Dense-attention models with more non-expert weights per layer benefit more.
### Results (current fleet — Jun 30, 2026 stress-test benchmark)

| Model | Quant | Ctx | Load | Gen t/s | Gen2 t/s | PP t/s | VRAM | Free |
|-------|-------|:---:|:----:|:-------:|:--------:|:------:|:----:|:----:|
| Qwythos-9B-MTP-Q6 | Q6_K | 131K | 0s¹ | **95.2** | 81.9 | 567 | 10,558 | 445 |
| Ornith-9B-Q5 | Q5_K_M | 64K | 15s | **69.2** | 70.2 | 236 | 7,186 | 3,817 |
| Qwen3.6-35B-MTP | Q4_K_M | 230K | 60s | **60.0** | 59.1 | 65 | 10,936 | **67** |
| Gemma-26B-200K | Q4_K_XL | 230K | 36s | **56.7** | 57.9 | 109 | 10,502 | 501 |
| GLM-4.7-Flash | Q4_K_M | 200K | 47s | **30.4** | 30.8 | 72 | 10,332 | 671 |
| Ornith-35B-Q6-MTP | Q6_K | 131K | 65s | **21.2** | 25.1 | 33 | 9,894 | 1,109 |

¹ Already warm from prior test. All measurements via llama-swap on RTX 2080 Ti 11 GB (b9743 sm75), 500-token generation, q8_0 KV cache.

**Ornith details:** Ornith-1.0-35B is a `qwen35moe` MoE model (35B total, ~3B active) with MTP heads surgically grafted from a sibling finetune via the Frankenstein approach (skinnyctax). It uses `--spec-type draft-mtp --spec-draft-n-max 4` for self-speculative decoding. MTP acceptance: 95% at pos 1, 87% at pos 4, mean accepted length 4.7 tokens. n-cpu-moe 30 was the sweet spot (10/40 expert layers on GPU). n-cpu-moe 29 and 32 were slower; 28 OOM'd. `--spec-draft-n-max 4` beat 1/2/3 on this hardware.

See `references/ornith-35b-onboarding.md` for the full benchmark protocol.

¹ At ncm=42 + 128K ctx, VRAM = 8.4 GB (weights) + 6.3 GB (KV cache) = **14.7 GB total — OOMs** on 11 GB card. The ncm=46 row is the validated fix: 2 layers' MoE on GPU keeps ~2.5 GB for KV cache, fitting 128K context at q8_0. Measured: 33.7 t/s, 30K prompt tokens without OOM.

**VRAM column note:** Values shown are model-weights-only (not including KV cache) unless noted. Always add KV cache budget separately — see `inference/vram-estimation` for per-model KV cache formulas.

**Diminishing returns** (rest of paragraph unchanged):
- Each layer of experts ≈ 800 MB on GPU
- Models with more total/MoE ratio (e.g., 80B/3B active) have less non-expert weight per layer, so each layer freed from CPU moves proportionally less compute

## Pitfall: --spec-type Binary Mismatch

sm75 binary accepts: `none,draft,eagle3,mtp,...` (NOT `draft-mtp`)
Upstream binary accepts: `none,draft-simple,draft-eagle3,draft-mtp,...` (NOT `mtp`)

Using the wrong one = immediate crash on startup.

- **Three-way drift:** disk ↔ llama-swap ↔ Hermes IDs must match. Mismatch causes silent context truncation or switch failure.
- **🔴 CRITICAL: `--chat-template-kwargs` is DEPRECATED** — use `--reasoning off`. The deprecated flag silently routes output to `reasoning_content` instead of `content`, producing empty message bodies. ALL models must be checked. Verify: `grep -c "chat-template-kwargs" ~/.config/llama-swap/config.yaml` returns 0.
- **Always set `--spec-draft-type-k q8_0 --spec-draft-type-v q8_0` explicitly** on any model using `--spec-type mtp` or `--spec-type draft-mtp`. Draft has its own KV cache and must be set explicitly even when matching the main cache type.
- **MoE OOM trap:** lowering `-ngl` is wrong for MoE; use `--n-cpu-moe 128` instead.
- **Background processes outlive cancellation:** use foreground with generous timeout, or save session_id to kill explicitly.
- **MTP KV slot boundary bug (GitHub #23658):** draft acceptance can collapse at specific context sizes; adjust ctx by ±1024 if speed drops.
- **Small batch paradox on 11GB:** smaller batches (256/256) can outperform larger ones on VRAM-constrained GPUs.

### Fleet Inventory Reporting

When the user asks for "a list of models with full stats" (params, quant, disk, VRAM, context, speed, HF source):

```bash
# Full inventory with GGUF metadata
python3 ~/.hermes/skills/devops/local-model-fleet-management/scripts/gguf-metadata-scan.py

# Include speed/VRAM benchmark data (auto-collects from references/)
python3 ~/.hermes/skills/devops/local-model-fleet-management/scripts/gguf-metadata-scan.py --speed-data

# Quick disk inventory
ls -lhS /models/downloads/*.gguf | awk '{printf "%s  %s\n", $5, $9}'

# Current llama-swap registered models
curl -s :9292/v1/models | python3 -c "import sys,json; [print(f'{m[\"id\"]}: {m.get(\"context_length\",\"?\")} ctx') for m in json.load(sys.stdin).get('data',[])]"

# Disk usage summary
du -sh /models/downloads/ && df -h /models/
```

**Comprehensive table format (for reports to user):**

| Model | Params | Quant | Disk | VRAM | Ctx | t/s | HF Source |
|---|---|---|---|---|---|---|---|

**Speed benchmark data** (measured on RTX 2080 Ti 11GB via `llama-server`): Stored in `references/fleet-benchmarks-*.md`. The `--speed-data` flag auto-merges this into the GGUF metadata scan.

**HF Sources** — determined at download time. Common patterns:
| Pattern | Uploader | Examples |
|---------|----------|---------|
| `unsloth/<model>-GGUF` | Unsloth | Most Qwen3, GLM, Nemotron, GPT-OSS quants |
| `yuxinlu1/gemma4-v2` | yuxinlu1 | Gemma 4 QAT quants (12B, 26B, drafts) |
| `cerebras/<model>` | Cerebras | REAP variants |
| `zai-org/<model>` | Zhipu AI | GLM-4.7-Flash (non-REAP) |

**Context cross-reference:** Every model's `-c` value in llama-swap config must equal its `context_length` in Hermes custom_providers. Run:
```bash
python3 -c "
import yaml
with open('/home/rurouni/.hermes/config.yaml') as f:
    h = yaml.safe_load(f)
with open('/home/rurouni/.config/llama-swap/config.yaml') as f:
    l = yaml.safe_load(f)
for m in h['custom_providers'][0]['models']:
    print(f'{m[\"id\"]}: hermes_ctx={m.get(\"context_length\",\"?\")}')
"
```

### Fleet Redundancy Assessment Heuristics

When auditing the fleet, assess each model against these criteria:

| Criterion | Question to ask | Action if true |
|-----------|----------------|----------------|
| Role overlap | Does another model serve the same role (e.g., two 9B coders)? | Drop the weaker/less maintained one |
| Speed ratio | Is it >50% slower than the cover model for the same class? | Drop if no unique capability |
| Community validation | Is it an obscure merge/community fine-tune with sparse discussions? (<100 HF downloads/mo) | Prefer well-documented alternatives |
| Maintenance burden | Requires a custom fork (PrismML, etc.)? | Assess if the use case justifies fork lock-in |
| VRAM safety margin | Uses >90% of available VRAM? | Reduce ctx or quantization, or drop |
| Actual usage | Has it been used in the last 30 days? | If not, drop (TTL is not usage) |

**Full stress-test methodology** (research → community validation → scorecard → disk cleanup → verification): see `references/fleet-stress-test-methodology.md`. Run this when the user asks "audit the fleet", "stress test models", or "what should I keep/remove".

### Full Fleet Cleanup Procedure (Cross-System Sync)

When removing models, the change must propagate through **all** systems. Missing even one produces silent drift:

1. **Disk** — delete the GGUF file(s) from `/models/downloads/`
2. **llama-swap config** — remove the model's profile from `~/.config/llama-swap/config.yaml`
3. **Hermes custom_providers** — remove the model entry from `~/.hermes/config.yaml`
4. **Hermes fallback_providers** — remove stale references
5. **OpenCode config** — remove from `~/.config/opencode/opencode.json` under `provider.local-gateway.models`
6. **Open WebUI SQLite** — remove from both `model` table and `config` table (`openai.api_configs.0.model_ids` + `openai.model_ids`). See `references/openwebui-sqlite-model-list.md` for the bulk sync script.
7. **AGENTS.md** — update `## Local Models` section
8. **Memory** — update the fleet entry
9. **llama-swap restart** — apply changes

The reverse is true for **adding** a model: it must appear in all 8 locations + have a llama-swap restart.

**Cross-system audit script** (run any time you suspect drift):
```bash
python3 ~/.hermes/skills/devops/local-model-fleet-management/scripts/cross-ref-audit.py
```

```bash
# Step 1: Delete GGUF file(s)
rm -v /models/downloads/<model-file1>.gguf /models/downloads/<model-file2>.gguf

# Step 2: Remove from llama-swap config
python3 -c "
import yaml, os
cfg = os.path.expanduser('~/.config/llama-swap/config.yaml')
with open(cfg) as f: data = yaml.safe_load(f)
models = data.get('models', {})
stale = ['model-id-1', 'model-id-2']  # swap profile names
for name in stale:
    if name in models:
        del models[name]
        print(f'Removed: {name}')
data['models'] = models
with open(cfg, 'w') as f:
    yaml.dump(data, f, default_flow_style=False, width=120)
"

# Step 3: Remove from Hermes config (models + fallback_providers)
python3 -c "
import yaml, os
path = os.path.expanduser('~/.hermes/config.yaml')
with open(path) as f: data = yaml.safe_load(f)
stale_ids = {'model-id-1', 'model-id-2'}
for p in data.get('custom_providers', []):
    p['models'] = [m for m in p.get('models', []) if m['id'] not in stale_ids]
data['fallback_providers'] = [
    fb for fb in data.get('fallback_providers', [])
    if fb.get('model') not in stale_ids
]
# Update environment hint
env = data.get('environment_hint', '')
import re
data['environment_hint'] = re.sub(r'\d+ local models', lambda m: str(int(m.group(0).split()[0]) - n), env)
with open(path, 'w') as f:
    yaml.dump(data, f, default_flow_style=False, sort_keys=False, width=1000)
print('Hermes config cleaned')
"

# Step 4: Update AGENTS.md
# Edit ~/.hermes/AGENTS.md — update ## Local Models section

# Step 5: Update memory
# Use memory() to update fleet entry

# Step 6: Restart llama-swap
sudo systemctl restart llama-swap.service

# Step 7: Verify
curl -s :9292/v1/models | python3 -c "import sys,json; [print(m['id']) for m in json.load(sys.stdin).get('data',[])]"
ls /models/downloads/*.gguf
```

### Context Size Audit Procedure  
The fleet skill and memory reference fleet-audit-june-2026 already have this.

## Binary Update Audit Procedure

When asked whether llama.cpp binaries need updating, do NOT guess. Use this research procedure:

### 0. Check Build Environment First

Before building, verify:
```bash
# CUDA toolkit must be 12.x (12.6 confirmed working)
nvcc --version

# g++ version must be ≤13 for CUDA 12.x nvcc compatibility
g++-13 --version  # g++ 14 WILL fail even with --allow-unsupported-compiler

# The correct build command when g++ 14 is default:
cmake -B build -DGGML_CUDA=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-13 \
  -DCMAKE_C_COMPILER=/usr/bin/gcc-13 \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++-13
```

**Known nvcc + g++ failure:** CUDA 12.x nvcc cannot compile with g++ 14 headers due to `__type_pack_element` template resolution. The `--allow-unsupported-compiler` flag is NOT sufficient — the C++14/17 header templates from g++ 14 are structurally incompatible. Install g++-13 alongside g++ 14.

After building a dynamically-linked binary (`BUILD_SHARED_LIBS=ON`):
```bash
sudo cp build/bin/*.so* /usr/local/lib/
sudo ldconfig
```
The binary is ~18 KB because it links to runtime .so files.

### 1. Establish Current vs Latest
```bash
# Get current binary versions
/usr/local/bin/llama-server-sm75 --version 2>&1 | head -1
/usr/local/bin/llama-server-upstream --version 2>&1 | head -1

# Check latest release tag via GitHub API
curl -s "https://api.github.com/repos/ggml-org/llama.cpp/releases/latest" | python3 -c "
import sys, json; r = json.load(sys.stdin); print(f'Latest: {r[\"tag_name\"]}')
"
```

### 2. Quantify the Gap
```bash
# Compare our build vs master
curl -s "https://api.github.com/repos/ggml-org/llama.cpp/compare/<OUR_HASH>...master" | \
  python3 -c "import sys,json; d=json.load(sys.stdin); print(f'{d.get(\"ahead_by\",\"?\")} commits, {d.get(\"behind_by\",\"?\")} behind, {d.get(\"total_commits\",\"?\")} total')"

# Or use the compare URL directly
# Example: https://github.com/ggml-org/llama.cpp/compare/b9341...master
```

### 3. Search for Meaningful Changes
Not all 400+ commits matter. Search for relevant PRs:
```bash
# CUDA performance PRs
curl -s "https://api.github.com/search/issues?q=repo:ggml-org/llama.cpp+type:pr+is:merged+label:cuda+updated:><DATE_OF_OUR_BUILD>&per_page=10" | \
  python3 -c "import sys,json; [print(f'CUDA: #{i[\"number\"]}: {i[\"title\"][:100]}') for i in json.load(sys.stdin).get('items',[])]"

# Speculative decoding PRs (MTP/Eagle)
curl -s "https://api.github.com/search/issues?q=repo:ggml-org/llama.cpp+type:pr+is:merged+spec+in:title+updated:><DATE>&per_page=10" | ...

# Server PRs (functionality)
curl -s "https://api.github.com/search/issues?q=repo:ggml-org/llama.cpp+type:pr+is:merged+label:server+updated:><DATE>&per_page=5" | ...
```

### 4. Assess Impact by Category
| Change Type | Impact | Worth Rebuilding? |
|---|---|---|
| **MMVQ/Turing optimizations** (e.g., `MMVQ_PARAMETERS_TURING`) | 5-10% gen speed on RTX 2080 Ti | Mildly — if building anyway |
| **Eagle3 spec decoding** (e.g., Qwen3.5/3.6) | Better draft acceptance | If you use that model |
| **CUDA kernel fixes** (e.g., fattn overflow, col2im) | Stability fixes | Only if hitting the bug |
| **Server refactoring/features** (87 PRs in one gap) | Bugfixes, new endpoints | Low priority |
| **CI/infra changes** | Zero impact | Skip |

### 5. Read Release Notes for Important Tags
```bash
# Get release notes for the past N releases to spot meaningful changes
curl -s "https://api.github.com/repos/ggml-org/llama.cpp/releases?per_page=15" | \
  python3 -c "
import sys, json, re
releases = json.load(sys.stdin)
for r in releases:
    tag = r['tag_name']
    body = r.get('body', '')
    clean = re.sub(r'<[^>]+>', '', body).strip()
    # Extract non-download-link lines
    lines = [l.strip() for l in clean.split('\n') if l.strip() 
             and not l.startswith('macOS') and not l.startswith('Linux') 
             and not l.startswith('Windows') and not l.startswith('Android')
             and not l.startswith('openEuler') and not l.startswith('UI')
             and not l.startswith('- [')]
    notes = ' | '.join(lines[:2]) if lines else '(binaries)'
    print(f'{tag}: {notes[:160]}')
"
```

### 6. Build Decision
- If no CUDA/spec/performance PRs exist: **do not rebuild** — current builds are fine
- If Turing-specific optimizations exist: **worthwhile but not urgent** (5-10% gain)
- If Eagle3 or major spec decoding improvements exist for models in fleet: **worth building**
- Estimated build time: ~30 min per binary with CUDA

**No pre-built CUDA Linux binaries are provided** — must build from source with `cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server`.

### 7. Post-Build: Verify Binary Capabilities

After building a new binary, verify key features before deploying:

```bash
# Check version
/usr/local/bin/llama-server-sm75 --version 2>&1 | head -1

# Verify Eagle3 support
/usr/local/bin/llama-server-sm75 --help 2>&1 | grep "draft-eagle3"

# Check spec types
/usr/local/bin/llama-server-sm75 --help 2>&1 | grep "spec-type"

# Benchmark a known model to detect regressions
# gemma-12b should hit ~96 t/s on b9743+ (known regression from #24514)
```

**IMPORTANT: TurboQuant KV cache types (turbo3/turbo4) are NOT available in upstream master.**

The old sm75 binary was built from the `feat/mtp-turboquant-kv-cache` fork which added custom cache types: `turbo2, turbo3, turbo4`. These were NOT merged into upstream master (PR #23962 merged a different version that doesn't include the cache types). 

Upstream `--cache-type-k` / `--cache-type-v` only supports: `f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`.

If you previously relied on turbo cache types for VRAM-constrained models, you must either:
1. Reduce context size instead
2. Increase `-fitt` value
3. Accept higher VRAM usage with q8_0 cache

**Performance comparison (b9743 vs old b9341 mtp-turbo fork):**
| Metric | Old (b9341 fork) | New (b9743 upstream) | Delta |
|--------|-----------------|---------------------|-------|
| gemma-12b t/s | 117.8 | 96.2 | -18% (known regression) |
| qwen36-35b t/s | 45.8 | 51.4 | +12% |
| gemma4-v2 t/s | 54.0 | 52.7 | ~same |
| gemma4-v2 VRAM (ngl 99) | OOM | 7.7 GB fits all layers | Huge improvement |
| Cold start (gemma4-v2) | 25s | 3s | Much faster |
| Eagle3 support | ❌ | ✅ | New capability |

## Provider Cleanup: Free-Tier Only

When a user asks to show only free models from a provider:

### OpenRouter / OpenCode
1. Fetch the model list from the API (Unsloth HF API or OpenRouter models page)
2. Identify free models — they have `-free` suffix on OpenCode Zen, or `:free` suffix on OpenRouter, or `$0/M` pricing
3. **Strip the provider's model list** to only free entries:
   ```python
   cfg['providers'][name]['models'] = {k: v for k, v in models.items() if k.endswith('-free')}
   ```
4. For providers with **no free tier** (e.g., OpenCode Go), **remove the entire provider block**
5. Restart the gateway

### Known Free Models
| Provider | Free models |
|----------|-------------|
| OpenCode Zen | `deepseek-v4-flash-free`, `qwen3.6-plus-free`, `minimax-m3-free`, `mimo-v2.5-free`, `nemotron-3-ultra-free`, `north-mini-code-free` |
| OpenRouter | `nvidia/nemotron-3-ultra-550b-a55b:free`, `nvidia/nemotron-3-super-120b-a12b:free`, `poolside/laguna-m-1:free`, `cohere/north-mini-code:free`, `openai/gpt-oss-120b:free`, `google/gemma-4-31b-it:free` and 6 more |
| NVIDIA NIM | All 121 models free for development (rate-limited) |

### Pitfall: `model_id` format
- OpenRouter uses colon-separated format: `nvidia/nemotron-3-ultra-550b-a55b:free`
- OpenCode uses dash-suffix: `nemotron-3-ultra-free`
- The `:free` suffix is part of the model ID for OpenRouter routing

## Eagle3 Speculator Setup

Eagle3 speculative decoding requires a **speculator head** (small model, ~1 GB) paired with a **matching base model**. The speculator is architecture-locked — it will NOT work with any other base model.

### Workflow

1. **Download the speculator** (safetensors format, ~327 MB to 1 GB):
   ```bash
   mkdir -p /models/downloads/eagle3-speculator
   curl -L -o /models/downloads/eagle3-speculator/model.safetensors "<url>"
   curl -sL -o /models/downloads/eagle3-speculator/config.json "<url>"
   ```

2. **Download the matching base model** as GGUF (same architecture, same size):
   ```bash
   # Find matching GGUF from unsloth or bartowski
   curl -s "https://huggingface.co/api/models?search=<MODEL-NAME>-GGUF&sort=downloads" | python3 -c "import sys,json; [print(m['modelId']) for m in json.load(sys.stdin)[:5]]"
   ```

3. **Convert the speculator to GGUF** (requires the base model's HF config for vocabulary):
   ```bash
   # First download the base model's config files
   pip install huggingface_hub -q
   python3 -c "
   from huggingface_hub import snapshot_download
   snapshot_download('<base-model-org>/<base-model-name>',
       local_dir='/tmp/base-model-config',
       allow_patterns=['tokenizer.json', 'tokenizer_config.json', 'config.json'])
   "

   # Then convert
   python3 convert_hf_to_gguf.py \
     /models/downloads/eagle3-speculator \
     --outfile /models/downloads/eagle3-speculator/eagle3.gguf \
     --target-model-dir /tmp/base-model-config
   ```
   - **--target-model-dir** (NOT `--target-model`) — points to the base model's HF config directory, NOT the base model GGUF file
   - The converter needs the base model's config.json, tokenizer.json, and tokenizer_config.json for vocabulary metadata
   - No `--speculator` flag exists — the converter auto-detects Eagle3 from the model's config.json

### Constraints
- Most Eagle3 speculators are in **safetensors format** (NOT pre-built GGUFs) and require conversion
- Exception: `jdluzen/Qwen3-Coder-Next-Eagle3-GGUF` provides pre-built GGUFs, but they require a llama.cpp build that supports `qwen3next` Eagle3 architecture — NOT yet in mainline as of b9743 (the model's repo explicitly says \"not available in mainline llama.cpp yet\")
- The base model MUST match exactly (same architecture, same parameter count) — the speculator is architecture-locked
- The speculator is tiny (~1 GB at Q4_K_M or ~327 MB bf16) and loads quickly, runs entirely on GPU
- Expected speedup on benchmark hardware (8× GPU, tensor parallel): ~50% over non-speculative inference
- **On single GPU with CPU expert offload (`--n-cpu-moe`), Eagle3 gives marginal improvement** (~1-3 t/s) because the bottleneck is PCIe bandwidth for expert weights, not token prediction speed
   --spec-draft-model /models/downloads/eagle3-speculator/eagle3-speculator.gguf\n   ```\n\n## Multi-Part GGUF Files (Unsloth Dynamic)

Some models (especially UD quants) are released as **split GGUFs**:
```
UD-Q3_K_M/Qwen3.5-122B-A10B-UD-Q3_K_M-00001-of-00003.gguf
UD-Q3_K_M/Qwen3.5-122B-A10B-UD-Q3_K_M-00002-of-00003.gguf
UD-Q3_K_M/Qwen3.5-122B-A10B-UD-Q3_K_M-00003-of-00003.gguf
```

### Handling
- **Download all parts** into the same directory with the original names
- **Do NOT rename the files** — llama.cpp finds continuation files by scanning the directory for the split naming pattern
- **Point llama-server to the FIRST file only** (`...00001-of-00003.gguf`) — it auto-discovers the rest
- **Verification:** `llama-server --model ...00001-of-00003.gguf --no-mmap -c 512` will load without "tensor data out of bounds" errors

### Pitfall: Partial Downloads
If a multi-part download is interrupted, parts may have mismatched sizes. Check:
```bash
# All parts should be within 1% of each other in size (for uniform splits)
ls -lh /path/to/*0000*-of-*.gguf
```
A part that's significantly smaller than others is incomplete.

### CDN Speed Differences
Download speed varies dramatically by HuggingFace account:
- **unsloth**: ~75-100 MB/s (fastest)
- **bartowski, mradermacher**: ~50-80 MB/s
- **lovedheart** (smaller orgs): ~9 MB/s — very slow
- **Qwen official**: ~40-70 MB/s

For slow downloads: use `notify_on_complete=true` + background with 7200s timeout. Check progress with `ls -lh` periodically. Do NOT kill and restart — curl's `-L -o` supports resume if the same filename is used.

**Known overlap:** `new-model-onboarding` skill exists but targets the old hardware setup (GTX 1080 Ti, systemd units, models.py script). It is **outdated** — this `local-model-fleet-management` skill is the current source of truth for the llama-swap based fleet on RTX 2080 Ti. The old skill references GTX 1080 Ti VRAM budgets, systemd-per-model service units, `models.py` scripts, and `hermes-config.path` watchers — none of which exist in the current llama-swap setup. It should be considered deprecated in favor of this skill.

## Pitfall: "Upstream Command Exited Prematurely" — VRAM Contention

When llama-swap reports `"upstream command exited prematurely"`, the root cause is often **not** a bad model file or wrong flags, but **leftover llama-server processes holding VRAM**.

### Debug Flow

1. **Check `nvidia-smi` first** — if free VRAM is <200 MiB, there's contention
2. **Check for stale llama-server processes:**
   ```bash
   pgrep -f "llama-server" | wc -l
   ```
3. **Kill all and retry:**
   ```bash
   pkill -9 -f llama-server
   sleep 3
   nvidia-smi --query-gpu=memory.free --format=csv,noheader  # should show 11002 MiB
   ```
4. **Then retry the model load**

### Example
This session: Mistral Nemo 12B failed to start with `-ngl 1`, `-c 256`, `--no-mmap` — everything minimal. Root cause: multiple test servers from prior benchmarks had consumed all 11GB. After `pkill -9 -f llama-server`, model loaded fine with `-ngl 99` at 80K context.

### Prevention
- Kill test servers explicitly after benchmarking (`pkill -f "<port>"`) before loading a different model
- Use `--no-host` to reduce per-server overhead
- Before debugging model flags, ALWAYS verify VRAM is free

When using `--n-cpu-moe`, the auto-fit tensor override **disables** automatic compute buffer sizing for the overridden layers. This means the model can appear to fit (weights + KV cache within VRAM) but still OOM on compute buffer allocation.

**Symptoms:**
```
E ggml_backend_cuda_buffer_type_alloc_buffer: allocating <N> MiB on device 0: cudaMalloc failed: out of memory
E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size <N>
E llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
```

**Root cause:** The `-fitt` value determines how much VRAM headroom to leave. With `--n-cpu-moe`, the auto-fit skips the overridden layers, so the compute buffer sizing is less accurate. A `-fitt` of 512 leaves only ~500 MB for compute buffers — fine without `--n-cpu-moe`, but insufficient when the override reduces headroom.

**Fix:** Increase `-fitt` to leave more headroom for compute buffers:
- `-fitt 512` → `-fitt 1024` (adds ~500 MB headroom)
- Or reduce `-c` to shrink the KV cache (e.g., 250K → 200K frees ~300 MB)

**Real example (qwen36-35b-mtp, RTX 2080 Ti 11 GB):**
| Context | --n-cpu-moe | -fitt | VRAM | Result |
|---------|-------------|-------|------|--------|
| 250K | 32 | 512 | ~9.2 GB | Compute buffer OOM (needs 781 MB) |
| 250K | 32 | 768 | ~9.2 GB | Still OOM |
| 200K | 32 | 1024 | ~9.2 GB | ✅ Works (compute buffer fits in ~1 GB headroom) |

The KV cache freed by reducing context (250K→200K) plus the increased `-fitt` together provide enough headroom. Either change alone may not suffice.

## Open WebUI Model List Management

Open WebUI stores model IDs in **three separate locations inside the same config JSON** (see `references/openwebui-sqlite-model-list.md`):

1. **`config` table → `data` JSON → `openai.api_configs["0"].model_ids`** — Controls the UI model picker dropdown. This is what the user sees.
2. **`config` table → `data` JSON → `openai.model_ids`** — A separate redundant list that also affects model visibility. This is EASILY FORGOTTEN because it's buried several keys deeper than `api_configs`.
3. **`model` table** — Per-user model registry (determines UI visibility, includes user-specific model entries added through the UI).

**ALL THREE must be updated** when models are added to or removed from llama-swap. The `model` table is the primary gate for UI visibility, but `openai.model_ids` operates as a secondary filter that can silently hide models even when `api_configs` and `model` table are correct.

**Common failure mode:** `api_configs.model_ids` shows the right models in the DB but `openai.model_ids` still has stale entries. The model picker shows old models and hides new ones. Fix both.

**⚠️ If Open WebUI crashes on startup with `TypeError: fromisoformat: argument must be str`**, see `references/openwebui-sqlite-datetime-crash.md` — the `config` table's `updated_at` column has an integer instead of a datetime string.

**Known Open WebUI SQLite corruption: `updated_at` as integer** — Open WebUI can write an integer Unix timestamp to the `config.updated_at` column (e.g., `1782594088`) instead of a datetime string (`2026-06-04 19:32:30`). SQLAlchemy's `str_to_datetime` processor calls `.fromisoformat()` on the value, which crashes with `TypeError: fromisoformat: argument must be str`. This prevents Open WebUI from starting (restart loop). Fix:
```sql
sqlite3 /path/to/webui.db
UPDATE config SET updated_at = datetime(updated_at, 'unixepoch') WHERE typeof(updated_at) = 'integer';
```

### Quick Add (Manual)
```bash
docker exec open-webui python3 -c "
import sqlite3, time
conn = sqlite3.connect('/app/backend/data/webui.db')
uid = conn.execute(\"SELECT id FROM user WHERE role='admin' LIMIT 1\").fetchone()[0]
now = int(time.time())
for mid in ['<model-id-1>', '<model-id-2>']:
    if not conn.execute('SELECT id FROM model WHERE base_model_id=?',(mid,)).fetchone():
        conn.execute('INSERT INTO model VALUES (?,?,?,?,?,?,?,?,1)', (f'model-{mid}', uid, mid, mid, '{}', '{}', now, now))
        print(f'Added: {mid}')
conn.commit()
"
docker restart open-webui
```

### Bulk Sync Script
See `references/openwebui-sqlite-model-list.md` → "Bulk Sync" section for a script that matches the Open WebUI model list to llama-swap's current fleet exactly.

### Config Location
Open WebUI runs as a Docker container (ghcr.io/open-webui/open-webui:latest, port 3081), with DB at `/app/backend/data/webui.db`.

## CDN Speed Differences (HF downloads)
Download speed varies dramatically by HuggingFace account:
- **unsloth**: ~75-100 MB/s (fastest)
- **bartowski, mradermacher**: ~50-80 MB/s
- **lovedheart** (smaller orgs): ~9 MB/s — very slow
- **Qwen official**: ~40-70 MB/s

For slow downloads: use `notify_on_complete=true` + background with 7200s timeout using the `hf` CLI (replaces deprecated `huggingface-cli`). Check progress with `ls -lh` periodically. Do NOT kill and restart — `hf download` supports resume.

### DiffusionGemma (PR #24423)
- Architecture: `diffusion_gemma` — NOT supported by `llama-server` (HTTP)
- Only has `llama-diffusion-cli` (CLI) and `llama-diffusion-gemma-server` (stdin/stdout protocol)
- Config uses **env vars** not CLI flags: `NGL` (GPU layers), `MAXTOK` (context), `FA` (flash attention)
- `--n-cpu-moe 128` works with the CLI for MoE offload, but NOT with the server
- Q4_K_M = 16 GB file, needs 18 GB total memory — runs at ~8 tok/s with offload on 11 GB VRAM
- To build: checkout PR #24423, build `llama-diffusion-cli` target
- When nvcc fails with g++-14: use `-DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-13`
- The PR is still a Draft — no HTTP server support yet

### Forced Context Extension
When a model's native context is too small (e.g., Moonlight at 8K), forced extension is possible but degrades quality:
```bash
# Required for models without YaRN scaling in GGUF metadata
--override-kv <arch>.context_length=int:<desired_ctx>
--rope-scaling linear --rope-freq-scale <factor>
```
Example for Moonlight (deepseek2, 8K native → 32K):
```
--override-kv deepseek2.context_length=int:32768
--rope-scaling linear --rope-freq-scale 4.0
```
**WARNING:** Linear scaling at 4x on a model not trained for extended context produces garbage output (token repetition). Only use if quality is not critical or the model documentation explicitly supports it.

The `context_length` in Hermes `custom_providers` must match the actual `-c` value in llama-swap config for every model. Mismatch causes silent context truncation: Hermes thinks the model supports 256K but llama-swap only allocated 131K.

**How to audit:**

```bash
# 1. Extract -c values from llama-swap config
grep -E '^\s+-c [0-9]+' ~/.config/llama-swap/config.yaml | while read line; do
  echo "$line"
done

# 2. Extract context_length values from Hermes config
python3 -c "
import yaml
with open('/home/rurouni/.hermes/config.yaml') as f:
    c = yaml.safe_load(f)
for m in c['custom_providers'][0]['models']:
    print(f'{m[\"id\"]}: {m[\"context_length\"]}')
"

# 3. Cross-reference: each model's -c must equal its context_length
# Verify the model IDs match too (llama-swap model key == custom_provider model id)
```

**Common mismatches found in the wild:**
- `gemma-12b`: llama-swap uses `-c 131072`, config had `256000` — double the actual capacity
- `gemma-26b`: llama-swap uses `-c 200000`, config had `256000` — overshoots VRAM headroom
- `deepseek-v2-lite`: entirely missing from Hermes config despite being in llama-swap

**Fix pattern when `hermes config set` fails** (stores arrays as JSON strings):
```python
import yaml, os
path = os.path.expanduser("~/.hermes/config.yaml")
with open(path) as f:
    data = yaml.safe_load(f)
# Fix models array (index 0 for first custom_provider)
data["custom_providers"][0]["models"] = <correct-list>
with open(path, "w") as f:
    yaml.dump(data, f, default_flow_style=False, sort_keys=False, allow_unicode=True, width=120)
```

```bash
# Verify the update didn't corrupt the YAML
python3 -c "import yaml; yaml.safe_load(open('/home/rurouni/.hermes/config.yaml'))" && echo "Valid YAML"
```

**⚠️ CRITICAL: When writing Hermes config via Python, use EXACTLY these kwargs:**
```python
yaml.dump(data, f, default_flow_style=False, sort_keys=False, width=1000)
```
A typo like `default_flow_state` instead of `default_flow_style` will **silently write an empty file**, destroying the config. Always validate the YAML after writing. Keep a recent backup accessible for quick restore:
```bash
# Restore from latest backup
cp ~/.hermes/config.yaml.bak-* ~/.hermes/config.yaml
```

### Hermes Config Corruption Recovery

If the config file is emptied (e.g., by a failed `yaml.dump()` with wrong kwargs):

1. **Check if the file is empty:** `wc -c ~/.hermes/config.yaml`
2. **Restore from latest backup:**
   ```bash
   ls -la ~/.hermes/config.yaml.bak-*  # find most recent
   cp ~/.hermes/config.yaml.bak-20260619-223108 ~/.hermes/config.yaml
   ```
3. **Or restore from a state snapshot:**
   ```bash
   ls -la ~/.hermes/state-snapshots/*/config.yaml
   cp ~/.hermes/state-snapshots/latest/config.yaml ~/.hermes/config.yaml
   ```
4. **Re-apply changes that were lost** (new models, provider edits) — use `python3 << 'PYEOF'` with the correct kwargs
5. **Validate after write:**
   ```bash
   python3 -c "import yaml; yaml.safe_load(open('/home/rurouni/.hermes/config.yaml'))" && echo "Valid YAML"
   ```
6. **Restart gateway:** `kill $(pgrep -f 'hermes.*gateway.run')` then restart via background
7. **Always verify gateway is responding:** `curl -s :18789/v1/models` should return `{"error":{"message":"Invalid API key"...}}` (key-required = alive) not `Connection refused` (dead)

**Pitfall:** After restoring from a snapshot, model IDs may be stale (e.g., `gemma-26b` vs `gemma-26b-200k`). Check and fix any drift between llama-swap model keys and Hermes model IDs.

## P0 Flag Verification Checklist

After any config change to all models, run:

```bash
grep -c "chat-template-kwargs" ~/.config/llama-swap/config.yaml     # must be 0
grep -c "reasoning off" ~/.config/llama-swap/config.yaml             # must equal model count (except Qwen — see pitfall)
grep -c "spec-draft-type-k" ~/.config/llama-swap/config.yaml         # must cover all MTP/draft-mtp models
grep "min-p 0 " ~/.config/llama-swap/config.yaml                     # must be empty (min-p=0 is broken)
grep -c "no-host -fitt" ~/.config/llama-swap/config.yaml             # should match model count
grep "mlock\\|no-mmap" ~/.config/llama-swap/config.yaml | head -3     # --no-mmap requires --mlock
```

**Pitfall: `reasoning_content` field vs `content` field.** GLM-4.7-flash-reap natively outputs to `reasoning_content` instead of `content`, leaving `content` empty. This is NOT caught by any of the above checks. To verify GLM works:

```bash
# Test that GLM produces non-empty content
curl -s :9292/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"glm-4.7-flash-reap","messages":[{"role":"user","content":"say hi"}],"max_tokens":20}' | \
  python3 -c "import sys,json; m=json.load(sys.stdin)['choices'][0]['message']; print('content:', m.get('content','')); print('reasoning:', m.get('reasoning_content',''))"
# content must not be empty — if empty, add --reasoning off to the model's cmd
```

**Known fact:** `--reasoning off` does NOT suppress Qwen models' native `<think>` tags. The check above will show Qwen models still emitting thinking tags despite the flag. This is expected — the flag prevents the server from ADDITIONALLY injecting reasoning but cannot strip model-native output. Open WebUI's Thinking toggle is the only way to suppress display.

## Model Thinking/Reasoning Behavior Reference

Some models natively output reasoning/thinking content regardless of `--reasoning off`. This affects how they behave in Open WebUI, API clients, and chat history.

### Which Models Produce Native Thinking

| Model | Arch | Native Thinking | Manifestation | `--reasoning off` works? |
|---|---|---|---|---|
| **gemma-26b-200k** | Gemma 4 MoE | No | Clean output | N/A |
| **glm-4.7-flash** | GLM MoE | **Yes** | Outputs `reasoning_content` field, `content` is empty | ✅ Fully — REQUIRED |
| **qwen36-35b-mtp** | Qwen 3.6 MoE | **Yes** | Native `<think>`/`</think>` tags in response | ❌ Tags persist |
| **ornith-35b-q6-mtp** | Qwen3.5 MoE | Yes | `<think>` tags in response | Falls back to normal — stripping works |
| **ornith-9b-q6** | Qwen3.5 Dense | **Yes** | `<think>` tags in response | Works — keep enabled; empty responses were OOM, not tag issue |
| **qwythos-9b-mtp-q6** | Qwen3.5 Dense | Yes | `<think>` tags in response (Claude Mythos trace format) | Works |

### ⚠️ Dense Reasoning Models: KV Cache OOM Masquerading as Tag Issue

**Initial suspicion:** `--reasoning off` on dense models (Ornith-9B) causes intermittent empty responses in Open WebUI.
**Actual root cause after investigation:** **KV cache OOM.** At `-c 131072` without `--no-kv-offload`, the KV cache allocates ~4.6 GB (36 layers × 4 KV heads × 128 hidden × 2 × 128K tokens × q8_0 byte). Combined with 6.9 GB weights = ~11.5 GB → exceeds 11 GB RTX 2080 Ti. The server crashes mid-stream (upstream disconnection), leaving Open WebUI with empty content.

**Evidence:**
- Streaming with `--reasoning off`: 0 reasoning events, 257 content events, clean completion
- Streaming without `--reasoning off`: 185 reasoning events + 9 content events per "hello" — excessive thinking tokens
- Direct API calls work fine; only Docker-streaming via Open WebUI shows the crash (because server dies mid-stream)
- llama-swap logs show `recovered from upstream disconnection during streaming` + `no valid JSON data found in stream`

**Fix (NOT removing `--reasoning off`):**
1. **Reduce context** to 32768 (frees ~3.5 GB KV cache → fits in 11 GB)
2. **Add `--no-kv-offload`** (move KV cache to system RAM)
3. **Keep `--reasoning off`** — works fine on dense models; empty responses were OOM crashes, not tag stripping
4. **Do NOT remove `--reasoning off`** — without it, model outputs 185+ thinking tokens per request, filling KV cache and slowing responses

**For MoE models** (ornith-35b, qwen36-35b): Same KV cache risk applies at high context. `--no-kv-offload` is always recommended for 11 GB cards with >64K context.

### Testing a Model for Native Thinking

```bash
curl -s :9292/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"<model-id>","messages":[{"role":"user","content":"say hi"}],"max_tokens":50}' | \
  python3 -c "
import sys,json
d=json.load(sys.stdin)
m=d['choices'][0]['message']
print('content:', repr(m.get('content','')))
print('reasoning:', repr(m.get('reasoning_content','')))
"
```
- If `reasoning_content` is non-empty → model has native thinking in `reasoning_content` field
- If `content` contains `<think>` or `<thinking>` tags → Qwen-style native thinking
- If both are clean → no native thinking

### Open WebUI Interaction

See `references/openwebui-thinking-toggle.md` for the critical interaction between Open WebUI's Thinking toggle and model routing. In short: **turning the toggle ON can cause Open WebUI to auto-select a reasoning-capable model** even when the user has a different model selected in the dropdown.

### GLM-4.7-flash-reap Specifics

- **Must** have `--reasoning off` — without it, the model outputs ONLY `reasoning_content` (empty `content` = invisible response)
- OOM fix at 131K q8_0 KV cache on 11 GB: increase `--n-cpu-moe` from 20 to 30 (frees ~0.8 GB, -20% t/s)
- Alternative: drop KV to q4_0 (keeps speed, small quality loss)
- Benchmark: 42.7 t/s at q8_0/q8_0 with n-cpu-moe 30; 53.5 t/s at q4_0/q4_0 with n-cpu-moe 20

Full detail in `references/` (20 files covering vram-table, hf-model-research, system-tuning, service-lifecycle, routing, cost-analysis, openwebui-thinking-toggle, and more). Newest: `references/2026-06-27-model-onboarding-ornith-agentworld.md` — Ornith-9B, Qwen-AgentWorld-35B configs, and Ornith-35B-Q8 n-cpu-moe sweep data.