---
name: local-model-fleet-management
description: "Manage your local LLM inference fleet — VRAM-conscious service lifecycle (systemd, GGUF alignment, path watchers) combined with smart model selection and routing strategy for cost-effective task dispatch."
version: 1.0.0
tags: [model-management, systemd, llama-server, vram, model-routing, cost-management]
---

# Local Model Fleet Management

## When to Use

- Setting up or modifying the local inference stack (llama-swap, systemd, model paths)
- Diagnosing model switch failures, cold load hangs, or VRAM contention
- Selecting a model for a task: local-first, escalate only when capability or speed demands
- Auditing three-way alignment: disk ↔ llama-swap config ↔ Hermes custom_providers
- GPU migration (e.g., Pascal → Turing) — re-benchmark every model afterward
- Binary update audit — checking whether llama.cpp builds need updating

## Core Workflow Rule: Research Before Building

The user has explicitly stated: **"stop guessing and creating things, always web research when you get stuck."**

This means:
- NEVER assume CLI flag names, build targets, or binary capabilities from memory alone.
- When uncertain about a flag, build target, or architecture support: consult the actual binary's `--help` output first, then web research (GitHub PR, HF model card, Unsloth docs).
- For model setup: read the HF model card README.md via raw URL before proposing flags.
- For build targets: check the GitHub PR's file tree and release notes before building.
- "Web research" means: GitHub issues/PRs, HF model cards, Unsloth docs, release notes — not asking the user to confirm guesses.
- **Quantization quality claims require benchmark data.** Do NOT claim one quantization is "better" or "similar quality" to another without citing actual perplexity, KL divergence, or downstream accuracy numbers from the technical report. The user will push back on unsubstantiated claims. If you don't have the numbers, say so and go find them — don't paraphrase the marketing summary.
- If a build fails (e.g., gcc version mismatch), research the fix (env var, flag, compiler switch) rather than guessing alternatives.

## Full Fleet Stress-Test Pipeline

When the user asks to "stress test", "audit", "bench test", or "scorecard" all models: see `references/fleet-stress-test-methodology.md` for the complete 7-phase pipeline (research → audit → redundancy assessment → benchmark → scorecard → cleanup → tweak audit).

The benchmark script at `scripts/bench-fleet-v2.py` runs a single pass per model through llama-swap: cold load timing, generation speed (500 tok, dual run), prompt processing speed, and VRAM capture. Run it any time the fleet changes or the user asks for fresh numbers:
```bash
python3 -u ~/.hermes/skills/devops/local-model-fleet-management/scripts/bench-fleet-v2.py
```

### Quant Upgrade Trade-off Analysis

When evaluating whether a lower quantization (Q6→Q5, Q8→Q6) can fund a context increase or faster inference:

1. **Get file sizes** for both quants from the HF repo
2. **Calculate VRAM delta:** dense models ∆VRAM ≈ ∆file_size; MoE models with --n-cpu-moe save proportionally less (only non-expert layers plus expert layers moved to GPU)
3. **KV cache budget:** `2 × n_layers × n_kv_heads × head_dim × target_ctx` bytes at q8_0
4. **Ceiling check:** total must be < 10,500 MiB on 11 GB card (including 500 MiB overhead)

**Real example (Ornith-9B, RTX 2080 Ti):** Q6→Q5 freed ~0.5 GB VRAM, enabling 32K→64K context with on-GPU KV cache, eliminating CPU KV bottleneck.

### sm75 Quantization Speed Hierarchy (RTX 2080 Ti)

**Counterintuitive finding confirmed July 2026:** On sm75 (Turing), the quantization *format* matters more than the *file size* for inference speed. Standard K-quants significantly outperform I-quants and Dynamic 2.0 UD quants at comparable bitrates.

**Benchmark data (Qwen3.6-35B-A3B on RTX 2080 Ti 11 GB):**
| Quant | Size | Format | Warm Gen (t/s) | VRAM | Load |
|---|---|---|---|---|---|
| **Q4_K_M (Unsloth)** | 22 GB | K-quant | **56.5** | 10,874 | 61s |
| **APEX I-Compact MTP** | 17.3 GB | I-quant | 51.8 | 9,710 | 10s |
| **UD-IQ4_NL (Unsloth)** | 18.5 GB | Dynamic IQ4 | **38.7** | 10,036 | 11s |

**Analysis:**
- APEX I-Compact loses ~5 t/s (9%) vs Q4_K_M despite being 5 GB *smaller* — the I-quant's non-uniform bit layout doesn't align with sm75's preferred compute patterns
- UD-IQ4_NL loses ~18 t/s (32%) vs Q4_K_M — Dynamic 2.0's custom tensor type assignments have significant compute overhead on Turing
- The tradeoff is worth it when the smaller file allows more context or faster cold load (ornith-35b went from 24.8 to 28.5 t/s on APEX because expert swapping was the *real* bottleneck)
- **Rule of thumb:** For MoE models bottlenecked by expert swapping (gen < 30 t/s, VRAM under 10,200 MiB), a smaller I-quant/UD file can improve speed. For dense models that fit entirely in VRAM already, K-quants are fastest.

**Decision matrix:**
| Scenario | Best quant type | Why |
|---|---|---|
| Dense model fits in VRAM | K-quant (Q5/Q6) | No swap penalty, fastest compute |
| MoE model, VRAM < 10 GB | I-quant (APEX) | Less swapping offsets compute tax |
| MoE model with free VRAM headroom | K-quant | No swap = no reason to take compute hit |
| Context-limited by KV cache | I-quant | Smaller file = more room for context |
| Fast cold load needed | I-quant | 5-6x faster load |

**Bottom line on quality claims:** Always verify against actual benchmark data before recommending "better" quants. The APEX technical report shows I-Compact (PPL 6.669) is slightly *worse* than standard Q4_K_M (PPL 6.610). And on sm75, it's also slower. Only recommend it when the smaller file size directly solves a VRAM constraint.

### APEX I-Quant Upgrades for MoE Models

APEX (Adaptive Precision for EXpert Models) is an importance-based quantization (I-quant) that allocates variable bit widths per tensor, prioritizing quality over uniform K-quant layouts. For MoE models on limited VRAM, APEX can be transformative.

**⚠️ Quality caveat:** APEX I-Compact is NOT better quality than standard Q4_K_M (PPL 6.669 vs 6.610). See `references/2026-07-03-apex-vs-ud-quality-comparison.md` for the actual benchmark rankings. The APEX advantage is smaller file size and MTP baked in, not quality. For better-than-Q4 quality on 11 GB, prefer Unsloth UD-IQ4_NL or UD-Q4_K_XL.

**Key tradeoff:** I-quants are ~5-10% slower per token than K-quants at equivalent bitrate on Turing (sm75) because of the non-uniform compute patterns. However, the MUCH smaller file size reduces CPU-GPU expert swapping, which is the actual bottleneck on 11 GB cards. The net effect is usually positive.

**APEX MTP GGUFs** (with MTP head baked into a single file):
- Ornith-35B: SC117/Ornith-1.0-35B-MTP-APEX-GGUF (I-Compact 17 GB)
- Qwen3.6-35B: mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF (I-Compact 17.3 GB, I-Mini 14.3 GB, I-Nano 11.7 GB)
- Requires recent llama.cpp with MTP support (bakes `--spec-type draft-mtp` in the GGUF via `nextn_predict_layers`)

**When to replace K-quant with APEX:**
- MoE model is bottlenecked by CPU-GPU expert swapping (check: gen < 30 t/s, VRAM < 10,200 MiB)
- Current file is >20 GB but only 11 GB VRAM
- APEX variant is ≥30% smaller at similar quality tier

**Real example (Ornith-35B on RTX 2080 Ti):**
| Metric | Q6_K-MTP (28 GB) | APEX-I-Compact (17 GB) | Δ |
|---|---|---|---|
| Cold load | 64s | 12s | **5.3x** |
| Gen2 (warm) | 24.8 t/s | 28.5 t/s | **+15%** |
| Context | 131K | 262K | **2x** |
| VRAM | 9,894 MiB | 9,900 MiB | ≈ |
| Disk | 28 GB | 17 GB | -11 GB |

**Real example (Qwen3.6-35B on RTX 2080 Ti):**
| Metric | Q4_K_M (22 GB) | APEX-I-Compact MTP (17.3 GB) | Δ |
|---|---|---|---|
| Cold load | 61s | 10s | **6x** |
| Gen (warm) | 56.5 t/s | 51.8 t/s | **-8%** (I-quant tax) |
| Context | 230K | 230K | ≈ |
| VRAM | 10,874 MiB | 9,710 MiB | **-1.2 GB** |
| Disk | 22 GB | 17.3 GB | -4.7 GB |

APEX on Qwen3.6 gives up ~5 t/s due to I-quant compute overhead on sm75, but gains 1.2 GB VRAM headroom and 6x faster cold loads. Quality is ≈ Q4_K_M (see quality comparison reference).

### Ornith-9B Dense → MTP Replacement

The Ornith-1.0-9B dense model from deepreinforce-ai can be upgraded to an MTP variant from protoLabsAI:

**Source:** https://huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP-GGUF

The GGUF bakes an MTP draft head on top of the same trunk. Required: `--spec-type draft-mtp`.

**Speed on RTX 2080 Ti (Q5_K_M):** Dense 69.7 t/s → MTP 71.3 t/s with `--spec-draft-n-max 2`. Higher n values (3+) cause regression on sm75 due to verify overhead.

Available quants: IQ2_M (3.9 GB), IQ3_M (4.7 GB), IQ4_XS (5.5 GB), Q4_K_M (5.8 GB), **Q5_K_M (6.2 GB)**, Q6_K (7.6 GB), Q8_0 (9.8 GB)

### Qwythos v3 Looping Fix

The Qwythos-9B-Claude-Mythos-5-1M model had a critical v3 release (Jun 19 2026) that fixed:
- Chat template for preserved reasoning & adaptive thinking
- **Looping during long generation traces**
- Agentic use in harnesses (OpenCode, Hermes, Claude Code)

**Detection:** If a Qwythos GGUF was downloaded before Jun 19 2026, it's v1/v2. The file size is identical between versions (7,617,818,464 bytes — only the embedded chat template changed), so you must re-download to get v3 even if the file looks the same on disk. A SHA256 comparison is the only way to distinguish versions without downloading.

**Source:** https://huggingface.co/empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF

### GLM-4.7-Flash Removal Rationale

GLM-4.7-Flash was removed from the fleet on Jul 3 2026 because:
- No MTP support in GGUF form (only vLLM has `--speculative-config.method mtp`)
- Slowest MoE model at 29.7 t/s (qwen36-35b does 50.9 t/s, ornith-35b APEX does 28.5 t/s but with 262K context)
- Largest disk footprint (18 GB) with no unique capability
- All agentic coding use cases covered by qwen36-35b-mtp and ornith-35b-mtp

### Per-Model Tweak Audit

After benchmarking, compare each model's current flags against model-card recommendations. Common issues found in the wild:
- **Temperature mismatch** — Gemma 4 recommended temp=1.0, often run at 0.2
- **MTP draft-n-max too high** — n=2 often outperforms n=3-5 on 12GB VRAM
- **`--reasoning off` missing** on GLM models → empty `content` field
- **`--no-kv-offload` left on** after context reduction freed VRAM
- **VRAM margin < 200 MiB** → compute buffer OOM risk, add `--no-warmup`
- **Missing `--repeat-penalty`** — default is 1.0 (disabled). Smaller models (≤9B dense) need **`--repeat-penalty 1.10`** plus `--repeat-last-n 256`. Larger MoE models (≥30B) need **`--repeat-penalty 1.05`** — 1.10 would suppress legitimate token diversity on bigger models. The fleet's older configs had this gap across all models.
- **GGUF architecture metadata causing model self-misidentification** — `general.architecture` in GGUF metadata (e.g., `qwen35`) can cause models to hallucinate being larger variants (e.g., a 9B model claiming to be 35B MoE). This is NOT a model bug — the model sees its own architecture tag and fills in the most common association from training data. Fix: (a) verify the actual file is the right quant/size, (b) if the system prompt tells the model what it is, update it with the correct name, or (c) add a system message correcting its self-identification.
- **`--top-k` too high on small dense models** — 9B dense models defaulting to top-k 20 repeat more often. Reduce to `--top-k 10` for ≤9B dense. MoE models at 35B+ can keep 20-40.

### Tweak → Re-Benchmark Workflow

After applying changes to model flags (temp, MTP draft length, context size, batch size, etc.), always re-benchmark to measure the impact. Use the same benchmark script for apples-to-apples comparison:

```bash
# Before tweaks
python3 -u ~/.hermes/skills/devops/local-model-fleet-management/scripts/bench-fleet-v2.py

# Apply tweaks, restart llama-swap
...

# After tweaks
python3 -u ~/.hermes/skills/devops/local-model-fleet-management/scripts/bench-fleet-v2.py
```

**What to look for in the before/after comparison:**
- **Gen t/s (cold):** first-request speed after cold load — affected by --no-warmup, batch size
- **Gen2 t/s (warm):** second-request speed — better measure of MTP draft acceptance, KV cache efficiency
- **PP t/s:** prompt processing — affected by context size, n-cpu-moe, batch/ubatch
- **VRAM/Free:** VRAM margin — anything <200 MiB free is a crash risk
- **MTP n=2 vs n=4:** warm gen often improves with n=2; cold gen may dip slightly
- **Temp change:** should not affect speed measurably, but large swings (>0.2 to >1.0) change output token distribution which can affect acceptance rates

**Real example from Jun 30 session (RTX 2080 Ti):**
| Model | Change | Before t/s | After t/s | Δ |
|-------|--------|:----------:|:---------:|:-:|
| Qwythos-9B | MTP n=4→2 | 95.2 | 86.6 | -9% cold |
| Qwythos-9B (warm) | MTP n=4→2 | 81.9 | **91.9** | **+12% warm** |
| Ornith-9B | 64K→96K ctx | 69.2 | **69.7** | **~same** (free context!) |

## Key Commands

- `systemctl status llama-swap.service` — verify the proxy is active
- `curl -s http://127.0.0.1:9292/v1/models | jq '.data[].id'` — list registered models
- `find /models/downloads/ -name '*.gguf' -maxdepth 2 | sort` — models on disk
- `grep -oP -- '-m \S+' ~/.config/llama-swap/config.yaml | while read f p; do [ -f "$p" ] && echo "✓ $p" || echo "✗ MISSING: $p"; done` — path audit
- `scripts/cross-ref-audit.py` — full three-way alignment check
- `nvidia-smi` — check VRAM availability before loading
- `cat /sys/kernel/mm/transparent_hugepage/enabled` — must show `madvise`

## New Model Integration Procedure

When adding a new GGUF model to the fleet, follow this checklist:

### 1. Research Architecture
```bash
# Get the original model config
curl -sL "https://huggingface.co/<org>/<model>/raw/main/config.json" | python3 -c "
import sys,json; d=json.load(sys.stdin)
for k in ['model_type','architectures','num_hidden_layers','num_attention_heads',
          'num_key_value_heads','hidden_size','max_position_embeddings',
          'num_experts','num_experts_per_tok']:
    if k in d: print(f'{k}: {d[k]}')
if 'text_config' in d: tc=d['text_config']; print(f'text_config.model_type: {tc.get(\"model_type\")}'); print(f'text_config.num_experts: {tc.get(\"num_experts\")}'); print(f'KV heads: {tc.get(\"num_key_value_heads\")}')
"
```

### 1b. Read the Model Card
Always read the HF model card README.md via raw URL for recommended flags and notes:
```bash
curl -sL "https://huggingface.co/<org>/<model>/raw/main/README.md" | head -200
```
Look for: recommended context size, sampling params (temp/top-p/top-k), architecture changes (`gemma4_unified` vs `gemma3`), MTP availability, and any special requirements ("needs recent llama.cpp").
### 1c. Parse GGUF Metadata from Raw File

Use the dedicated `scripts/gguf-metadata-scan.py` script to extract architecture, context length, layer count, head counts, quantization, and HF source from GGUF files. It handles GGUF v3 format (u64 key lengths), big tokenizer arrays (100K+ vocab), and partial buffers:

```bash
# Scan all models in the fleet
python3 ~/.hermes/skills/devops/local-model-fleet-management/scripts/gguf-metadata-scan.py

# Output: formatted Markdown table (TTY) or JSON (piped)
python3 scripts/gguf-metadata-scan.py | jq '.[] | {file, architecture, context_length, block_count, quant, disk_gb}'

# Scan specific files
python3 scripts/gguf-metadata-scan.py /models/downloads/Qwen3.6-35B*.gguf
```

Key differences from the `gguf` Python package:
- No Python `gguf` module dependency (which can hang on large files >5 GB)
- Reads only the metadata header (~2 MB), skips tensor data entirely
- Prints formatted table or JSON, suitable for fleet audits and inventory reports

### 2. Check Binary Compatibility
```bash
# Check if the model_type is supported by each binary
strings /usr/local/bin/llama-server-sm75 | grep -c "<model_type>"    # sm75 support count
strings /usr/local/bin/llama-server-upstream | grep -c "<model_type>"  # upstream support count
```
- If sm75 has 0 refs and upstream has 74+ (nemotron_h pattern), it's upstream-only
- If both have refs, either binary works
- For sm75: check if `--spec-type mtp` is needed (check `--help 2>&1 | grep spec-type`)
- For upstream: check if `--spec-type draft-mtp` is needed

### 3. Check MTP Support
- Search the GGUF filename for "MTP" 
- Check Unsloth docs or README for MTP mentions
- Dense Qwen3 models from mradermacher or bartowski typically do NOT have MTP heads
- Unsloth GGUF variants often include MTP

### 4. Get Real File Size
```bash
curl -sIL "https://huggingface.co/<org>/<model>/resolve/main/<file>.gguf" 2>&1 | grep -i x-linked-size
```
Divide by 1024^3 for GB. This gives real file size, not the HF API 0-byte bug.

### 5. Estimate VRAM Budget
**For dense models:** VRAM ≈ file_size * (ngl / total_layers) + KV_cache + 0.5GB overhead
**For MoE models with --n-cpu-moe 128:** VRAM ≈ 3GB (active) + KV_cache + 0.5GB overhead

KV cache per token (q8_0) = 2 * n_layers * n_kv_heads * (hidden_size / n_attention_heads) bytes

**Efficiency heuristic:** Models with very few KV heads (e.g., Nemotron-3-Nano with 2 KV heads) have extremely small KV caches and can handle massive context on limited VRAM. This is the first thing to check.

### 6. Multi-Platform Registration

After the model is verified in llama-swap, register it on all three platforms so it's selectable everywhere:

#### 6a. Hermes custom_providers
```python
import yaml, os
path = os.path.expanduser("~/.hermes/config.yaml")
with open(path) as f:
    data = yaml.safe_load(f)
for cp in data.get("custom_providers", []):
    if cp.get("name") == "Local LLM (llama-swap)":
        cp["models"].append({
            "context_length": <CTX>,
            "description": "<Short description>",
            "id": "<llama-swap-model-key>",
            "name": "<Human-readable name>"
        })
        break
with open(path, "w") as f:
    yaml.dump(data, f, default_flow_style=False, sort_keys=False, width=1000)
# Validate
python3 -c "import yaml; yaml.safe_load(open('/home/rurouni/.hermes/config.yaml'))" && echo "Valid"
```

#### 6b. OpenCode local-gateway
Edit `~/.config/opencode/opencode.json` and add a model entry under `provider.local-gateway.models`:
```json
"<model-id>": {
  "name": "<Human-readable name>",
  "limit": {
    "context": <CTX>,
    "output": 16384
  }
}
```

#### 6c. Open WebUI
Open WebUI points to llama-swap via `OPENAI_API_BASE_URL=http://host.docker.internal:9292/v1` and auto-discovers models from `/v1/models`. Restart the container to refresh the list:
```bash
docker restart open-webui
```
Wait 15s then verify at http://localhost:3081 (model picker should show the new entry).

### 7. Verify the Full Switch Path

### 7. OOM Escalation Path (DO NOT use q4_0 KV first)
If a model OOMs on load:
1. **Reduce context** first (e.g., 256K → 160K → 128K → 64K → 16K)
2. **Raise -fitt** from 512 to 768/1024 (increases safety margin)
3. **Reduce draft aggressiveness** (lower --spec-draft-n-max, raise --spec-draft-p-min) or remove --spec-draft-ngl
4. **Add --no-kv-offload** (moves KV cache to system RAM, slower but works)
5. **Only then** test turbo cache or q4_0 KV cache

**Exception: GLM-4.7-flash-reap.** At 131K context with q8_0 KV cache, GLM needs ~3.6 GB for KV alone. Two valid fix paths:
- **Path A (keep q8_0 quality):** increase `--n-cpu-moe` from 20 to 30 — frees ~0.8 GB by offloading 10 more MoE experts to CPU. Speed drops ~20% (53.5 → 42.7 t/s).
- **Path B (keep speed):** downgrade KV cache to `q4_0/q4_0` — frees ~1.8 GB. Minor quality impact on long-context retrieval. Speed stays at ~53.5 t/s.
**IMPORTANT:** GLM also requires `--reasoning off` — without it, the model only outputs `reasoning_content` (empty `content` field = no visible response).

## Known CUDA Regressions

See `references/cuda-regressions-known.md` for tracked upstream issues affecting RTX 2080 Ti performance: #24514 (25% perf drop b9301→b9305) and #24670 (draft-mtp not activating on Turing sm_75).

## Research Agent Hallucination Warning

Delegate_task subagents for web research **invent flags that don't exist**. Verified in June 2026:
- `--cuda-streams N` — does not exist on either llama.cpp binary
- Always verify proposed flags against actual binary --help output before applying

### Pitfall: `huggingface-cli` Deprecated (June 2026)

`huggingface-cli` is deprecated and no longer works. Use the new `hf` CLI instead:

```bash
# OLD (broken):
huggingface-cli download <org>/<repo> <file> --local-dir ./dir

# NEW:
hf download <org>/<repo> --include "<file>" --local-dir ./dir
```

The `hf` CLI is installed alongside `huggingface_hub`. Use `hf --help` for commands.
Key differences:
- Positional file args work directly: `hf download <org>/<repo> <file> --local-dir ./dir`
- `--local-dir-use-symlinks` does NOT exist on `hf` (omit it)
- Use `--force-download` to redownload cached files
- `--include` supports glob patterns (alternative to positional args)

## Pitfall: --poll and --n-cpu-moe Misunderstandings

From the binary --help output:
- `--poll <0...100>` — polling **level**, not milliseconds. Default 50. 30 recommended for single-user SM75.
- `--n-cpu-moe N` — keep the MoE weights of the **first N layers** in the CPU. NOT CPU thread count. NOT "expert threads".
- Setting n-cpu-moe too high (e.g., 128 on a 40-layer model) pushes all MoE experts to CPU. Safe but slow.
- Setting n-cpu-moe too low risks OOM if the model + KV cache exceed VRAM.

### --n-cpu-moe Performance Tuning

When a model has VRAM headroom, reducing `--n-cpu-moe` below the layer count moves expert weights to GPU, significantly improving speed:

**Methodology:**
1. Note current VRAM usage with `--n-cpu-moe 128` (all experts on CPU)
2. The model's layer count determines the max n-cpu-moe that keeps experts on CPU
   - `--n-cpu-moe N` where N >= layer_count = all experts on CPU
   - `--n-cpu-moe N` where N < layer_count = layers (N..end) experts on GPU
3. Try progressively lower values (e.g., 40 → 35 → 30 → 20) with `-c 16384` for fast testing
4. Check VRAM — if it fits in ~9-10 GB, benchmark to see if speed improved
5. If OOM, increase n-cpu-moe by 2-3

**Diminishing returns:** Models where experts are a small fraction of total parameters (e.g., coder-next-80b with 80B total / 3B active) gain less from moving a few layer experts to GPU because most compute time is still in the CPU-offloaded experts. Dense-attention models with more non-expert weights per layer benefit more.
### Results (current fleet — Jun 30, 2026 stress-test benchmark)

| Model | Quant | Ctx | Load | Gen t/s | Gen2 t/s | PP t/s | VRAM | Free |
|-------|-------|:---:|:----:|:-------:|:--------:|:------:|:----:|:----:|
| Qwythos-9B-MTP-Q6 (v3) | Q6_K MTP | 131K | 4s | **96.4** | 89.7 | 482 | 10,458 | 545 |
| Ornith-9B-MTP-Q5 | Q5_K_M MTP | 96K | 3s | **71.3** | 72.5 | — | 8,904 | 2,099 |
| Qwen3.6-35B-MTP | Q4_K_M MTP | 230K | 61s | **50.9** | 56.5 | 53 | 10,874 | 129 |
| Gemma-26B-200K | Q4_K_XL + draft | 230K | 41s | **51.5** | 55.8 | 118 | 10,502 | 501 |
| Ornith-35B-APEX-MTP | APEX-I-Compact MTP | 262K | **12s** | 22.8 | **28.5** | — | 9,900 | 1,103 |
| *(GLM-4.7-Flash removed Jul 3 — no MTP, slowest MoE, 18 GB)* |

¹ Already warm from prior test. All measurements via llama-swap on RTX 2080 Ti 11 GB (b9743 sm75), 500-token generation, q8_0 KV cache.

**⚠️ Tool Calling Correction (Jul 2026):** The fleet skill previously stated local models "do NOT support tool calling." This was wrong for `--jinja`-enabled models. 4 of 5 fleet models support OpenAI function/tool calling via llama.cpp `--jinja` flag. See `references/tool-calling-local-models.md` for test results, procedure, and per-model quirks.

**Ornith details:** Ornith-1.0-35B is a `qwen35moe` MoE model (35B total, ~3B active) with MTP heads surgically grafted from a sibling finetune via the Frankenstein approach (skinnyctax). It uses `--spec-type draft-mtp --spec-draft-n-max 4` for self-speculative decoding. MTP acceptance: 95% at pos 1, 87% at pos 4, mean accepted length 4.7 tokens. n-cpu-moe 30 was the sweet spot (10/40 expert layers on GPU). n-cpu-moe 29 and 32 were slower; 28 OOM'd. `--spec-draft-n-max 4` beat 1/2/3 on this hardware.

See `references/ornith-35b-onboarding.md` for the full benchmark protocol.

¹ At ncm=42 + 128K ctx, VRAM = 8.4 GB (weights) + 6.3 GB (KV cache) = **14.7 GB total — OOMs** on 11 GB card. The ncm=46 row is the validated fix: 2 layers' MoE on GPU keeps ~2.5 GB for KV cache, fitting 128K context at q8_0. Measured: 33.7 t/s, 30K prompt tokens without OOM.

**VRAM column note:** Values shown are model-weights-only (not including KV cache) unless noted. Always add KV cache budget separately — see `inference/vram-estimation` for per-model KV cache formulas.

**Diminishing returns** (rest of paragraph unchanged):
- Each layer of experts ≈ 800 MB on GPU
- Models with more total/MoE ratio (e.g., 80B/3B active) have less non-expert weight per layer, so each layer freed from CPU moves proportionally less compute

## Pitfall: --spec-type Binary Mismatch

sm75 binary accepts: `none,draft,eagle3,mtp,...` (NOT `draft-mtp`)
Upstream binary accepts: `none,draft-simple,draft-eagle3,draft-mtp,...` (NOT `mtp`)

Using the wrong one = immediate crash on startup.

- **Three-way drift:** disk ↔ llama-swap ↔ Hermes IDs must match. Mismatch causes silent context truncation or switch failure.
- **🔴 CRITICAL: `--chat-template-kwargs` is DEPRECATED** — use `--reasoning off`. The deprecated flag silently routes output to `reasoning_content` instead of `content`, producing empty message bodies. ALL models must be checked. Verify: `grep -c "chat-template-kwargs" ~/.config/llama-swap/config.yaml` returns 0.
- **Always set `--spec-draft-type-k q8_0 --spec-draft-type-v q8_0` explicitly** on any model using `--spec-type mtp` or `--spec-type draft-mtp`. Draft has its own KV cache and must be set explicitly even when matching the main cache type.
- **MoE OOM trap:** lowering `-ngl` is wrong for MoE; use `--n-cpu-moe 128` instead.
- **Background processes outlive cancellation:** use foreground with generous timeout, or save session_id to kill explicitly.
- **MTP KV slot boundary bug (GitHub #23658):** draft acceptance can collapse at specific context sizes; adjust ctx by ±1024 if speed drops.
- **Small batch paradox on 11GB:** smaller batches (256/256) can outperform larger ones on VRAM-constrained GPUs.

### Fleet Inventory Reporting

When the user asks for "a list of models with full stats" (params, quant, disk, VRAM, context, speed, HF source):

```bash
# Full inventory with GGUF metadata
python3 ~/.hermes/skills/devops/local-model-fleet-management/scripts/gguf-metadata-scan.py

# Include speed/VRAM benchmark data (auto-collects from references/)
python3 ~/.hermes/skills/devops/local-model-fleet-management/scripts/gguf-metadata-scan.py --speed-data

# Quick disk inventory
ls -lhS /models/downloads/*.gguf | awk '{printf "%s  %s\n", $5, $9}'

# Current llama-swap registered models
curl -s :9292/v1/models | python3 -c "import sys,json; [print(f'{m[\"id\"]}: {m.get(\"context_length\",\"?\")} ctx') for m in json.load(sys.stdin).get('data',[])]"

# Disk usage summary
du -sh /models/downloads/ && df -h /models/
```

**Comprehensive table format (for reports to user):**

| Model | Params | Quant | Disk | VRAM | Ctx | t/s | HF Source |
|---|---|---|---|---|---|---|---|

**Speed benchmark data** (measured on RTX 2080 Ti 11GB via `llama-server`): Stored in `references/fleet-benchmarks-*.md`. The `--speed-data` flag auto-merges this into the GGUF metadata scan.

**HF Sources** — determined at download time. Common patterns:
| Pattern | Uploader | Examples |
|---------|----------|---------|
| `unsloth/<model>-GGUF` | Unsloth | Most Qwen3, GLM, Nemotron, GPT-OSS quants |
| `yuxinlu1/gemma4-v2` | yuxinlu1 | Gemma 4 QAT quants (12B, 26B, drafts) |
| `cerebras/<model>` | Cerebras | REAP variants |
| `zai-org/<model>` | Zhipu AI | GLM-4.7-Flash (non-REAP) |

**Context cross-reference:** Every model's `-c` value in llama-swap config must equal its `context_length` in Hermes custom_providers. Run:
```bash
python3 -c "
import yaml
with open('/home/rurouni/.hermes/config.yaml') as f:
    h = yaml.safe_load(f)
with open('/home/rurouni/.config/llama-swap/config.yaml') as f:
    l = yaml.safe_load(f)
for m in h['custom_providers'][0]['models']:
    print(f'{m[\"id\"]}: hermes_ctx={m.get(\"context_length\",\"?\")}')
"
```

### Fleet Redundancy Assessment Heuristics

When auditing the fleet, assess each model against these criteria:

| Criterion | Question to ask | Action if true |
|-----------|----------------|----------------|
| Role overlap | Does another model serve the same role (e.g., two 9B coders)? | Drop the weaker/less maintained one |
| Speed ratio | Is it >50% slower than the cover model for the same class? | Drop if no unique capability |
| Community validation | Is it an obscure merge/community fine-tune with sparse discussions? (<100 HF downloads/mo) | Prefer well-documented alternatives |
| Maintenance burden | Requires a custom fork (PrismML, etc.)? | Assess if the use case justifies fork lock-in |
| VRAM safety margin | Uses >90% of available VRAM? | Reduce ctx or quantization, or drop |
| Actual usage | Has it been used in the last 30 days? | If not, drop (TTL is not usage) |

**Full stress-test methodology** (research → community validation → scorecard → disk cleanup → verification): see `references/fleet-stress-test-methodology.md`. Run this when the user asks "audit the fleet", "stress test models", or "what should I keep/remove".

### Full Fleet Cleanup Procedure (Cross-System Sync)

When removing models, the change must propagate through **all** systems. Missing even one produces silent drift:

1. **Disk** — delete the GGUF file(s) from `/models/downloads/`
2. **llama-swap config** — remove the model's profile from `~/.config/llama-swap/config.yaml`
3. **Hermes custom_providers** — remove the model entry from `~/.hermes/config.yaml`
4. **Hermes fallback_providers** — remove stale references
5. **OpenCode config** — remove from `~/.config/opencode/opencode.json` under `provider.local-gateway.models`
6. **Open WebUI SQLite** — remove from both `model` table and `config` table (`openai.api_configs.0.model_ids` + `openai.model_ids`). See `references/openwebui-sqlite-model-list.md` for the bulk sync script.
7. **AGENTS.md** — update `## Local Models` section
8. **Memory** — update the fleet entry
9. **llama-swap restart** — apply changes

The reverse is true for **adding** a model: it must appear in all 8 locations + have a llama-swap restart.

**Cross-system audit script** (run any time you suspect drift):
```bash
python3 ~/.hermes/skills/devops/local-model-fleet-management/scripts/cross-ref-audit.py
```

```bash
# Step 1: Delete GGUF file(s)
rm -v /models/downloads/<model-file1>.gguf /models/downloads/<model-file2>.gguf

# Step 2: Remove from llama-swap config
python3 -c "
import yaml, os
cfg = os.path.expanduser('~/.config/llama-swap/config.yaml')
with open(cfg) as f: data = yaml.safe_load(f)
models = data.get('models', {})
stale = ['model-id-1', 'model-id-2']  # swap profile names
for name in stale:
    if name in models:
        del models[name]
        print(f'Removed: {name}')
data['models'] = models
with open(cfg, 'w') as f:
    yaml.dump(data, f, default_flow_style=False, width=120)
"

# Step 3: Remove from Hermes config (models + fallback_providers)
python3 -c "
import yaml, os
path = os.path.expanduser('~/.hermes/config.yaml')
with open(path) as f: data = yaml.safe_load(f)
stale_ids = {'model-id-1', 'model-id-2'}
for p in data.get('custom_providers', []):
    p['models'] = [m for m in p.get('models', []) if m['id'] not in stale_ids]
data['fallback_providers'] = [
    fb for fb in data.get('fallback_providers', [])
    if fb.get('model') not in stale_ids
]
# Update environment hint
env = data.get('environment_hint', '')
import re
data['environment_hint'] = re.sub(r'\d+ local models', lambda m: str(int(m.group(0).split()[0]) - n), env)
with open(path, 'w') as f:
    yaml.dump(data, f, default_flow_style=False, sort_keys=False, width=1000)
print('Hermes config cleaned')
"

# Step 4: Update AGENTS.md
# Edit ~/.hermes/AGENTS.md — update ## Local Models section

# Step 5: Update memory
# Use memory() to update fleet entry

# Step 6: Restart llama-swap
sudo systemctl restart llama-swap.service

# Step 7: Verify
curl -s :9292/v1/models | python3 -c "import sys,json; [print(m['id']) for m in json.load(sys.stdin).get('data',[])]"
ls /models/downloads/*.gguf
```

### Context Size Audit Procedure  
The fleet skill and memory reference fleet-audit-june-2026 already have this.

## Binary Update Audit Procedure

When asked whether llama.cpp binaries need updating, do NOT guess. Use this research procedure:

### 0. Check Build Environment First

Before building, verify:
```bash
# CUDA toolkit must be 12.x (12.6 confirmed working)
nvcc --version

# g++ version must be ≤13 for CUDA 12.x nvcc compatibility
g++-13 --version  # g++ 14 WILL fail even with --allow-unsupported-compiler

# The correct build command when g++ 14 is default:
cmake -B build -DGGML_CUDA=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-13 \
  -DCMAKE_C_COMPILER=/usr/bin/gcc-13 \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++-13
```

**Known nvcc + g++ failure:** CUDA 12.x nvcc cannot compile with g++ 14 headers due to `__type_pack_element` template resolution. The `--allow-unsupported-compiler` flag is NOT sufficient — the C++14/17 header templates from g++ 14 are structurally incompatible. Install g++-13 alongside g++ 14.

After building a dynamically-linked binary (`BUILD_SHARED_LIBS=ON`):
```bash
sudo cp build/bin/*.so* /usr/local/lib/
sudo ldconfig
```
The binary is ~18 KB because it links to runtime .so files.

### 1. Establish Current vs Latest
```bash
# Get current binary versions
/usr/local/bin/llama-server-sm75 --version 2>&1 | head -1
/usr/local/bin/llama-server-upstream --version 2>&1 | head -1

# Check latest release tag via GitHub API
curl -s "https://api.github.com/repos/ggml-org/llama.cpp/releases/latest" | python3 -c "
import sys, json; r = json.load(sys.stdin); print(f'Latest: {r[\"tag_name\"]}')
"
```

### 2. Quantify the Gap
```bash
# Compare our build vs master
curl -s "https://api.github.com/repos/ggml-org/llama.cpp/compare/<OUR_HASH>...master" | \
  python3 -c "import sys,json; d=json.load(sys.stdin); print(f'{d.get(\"ahead_by\",\"?\")} commits, {d.get(\"behind_by\",\"?\")} behind, {d.get(\"total_commits\",\"?\")} total')"

# Or use the compare URL directly
# Example: https://github.com/ggml-org/llama.cpp/compare/b9341...master
```

### 3. Search for Meaningful Changes
Not all 400+ commits matter. Search for relevant PRs:
```bash
# CUDA performance PRs
curl -s "https://api.github.com/search/issues?q=repo:ggml-org/llama.cpp+type:pr+is:merged+label:cuda+updated:><DATE_OF_OUR_BUILD>&per_page=10" | \
  python3 -c "import sys,json; [print(f'CUDA: #{i[\"number\"]}: {i[\"title\"][:100]}') for i in json.load(sys.stdin).get('items',[])]"

# Speculative decoding PRs (MTP/Eagle)
curl -s "https://api.github.com/search/issues?q=repo:ggml-org/llama.cpp+type:pr+is:merged+spec+in:title+updated:><DATE>&per_page=10" | ...

# Server PRs (functionality)
curl -s "https://api.github.com/search/issues?q=repo:ggml-org/llama.cpp+type:pr+is:merged+label:server+updated:><DATE>&per_page=5" | ...
```

### 4. Assess Impact by Category
| Change Type | Impact | Worth Rebuilding? |
|---|---|---|
| **MMVQ/Turing optimizations** (e.g., `MMVQ_PARAMETERS_TURING`) | 5-10% gen speed on RTX 2080 Ti | Mildly — if building anyway |
| **Eagle3 spec decoding** (e.g., Qwen3.5/3.6) | Better draft acceptance | If you use that model |
| **CUDA kernel fixes** (e.g., fattn overflow, col2im) | Stability fixes | Only if hitting the bug |
| **Server refactoring/features** (87 PRs in one gap) | Bugfixes, new endpoints | Low priority |
| **CI/infra changes** | Zero impact | Skip |

### 5. Read Release Notes for Important Tags
```bash
# Get release notes for the past N releases to spot meaningful changes
curl -s "https://api.github.com/repos/ggml-org/llama.cpp/releases?per_page=15" | \
  python3 -c "
import sys, json, re
releases = json.load(sys.stdin)
for r in releases:
    tag = r['tag_name']
    body = r.get('body', '')
    clean = re.sub(r'<[^>]+>', '', body).strip()
    # Extract non-download-link lines
    lines = [l.strip() for l in clean.split('\n') if l.strip() 
             and not l.startswith('macOS') and not l.startswith('Linux') 
             and not l.startswith('Windows') and not l.startswith('Android')
             and not l.startswith('openEuler') and not l.startswith('UI')
             and not l.startswith('- [')]
    notes = ' | '.join(lines[:2]) if lines else '(binaries)'
    print(f'{tag}: {notes[:160]}')
"
```

### 6. Build Decision
- If no CUDA/spec/performance PRs exist: **do not rebuild** — current builds are fine
- If Turing-specific optimizations exist: **worthwhile but not urgent** (5-10% gain)
- If Eagle3 or major spec decoding improvements exist for models in fleet: **worth building**
- Estimated build time: ~30 min per binary with CUDA

**No pre-built CUDA Linux binaries are provided** — must build from source with `cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server`.

### 7. Post-Build: Verify Binary Capabilities

After building a new binary, verify key features before deploying:

```bash
# Check version
/usr/local/bin/llama-server-sm75 --version 2>&1 | head -1

# Verify Eagle3 support
/usr/local/bin/llama-server-sm75 --help 2>&1 | grep "draft-eagle3"

# Check spec types
/usr/local/bin/llama-server-sm75 --help 2>&1 | grep "spec-type"

# Benchmark a known model to detect regressions
# gemma-12b should hit ~96 t/s on b9743+ (known regression from #24514)
```

**IMPORTANT: TurboQuant KV cache types (turbo3/turbo4) are NOT available in upstream master.**

The old sm75 binary was built from the `feat/mtp-turboquant-kv-cache` fork which added custom cache types: `turbo2, turbo3, turbo4`. These were NOT merged into upstream master (PR #23962 merged a different version that doesn't include the cache types). 

Upstream `--cache-type-k` / `--cache-type-v` only supports: `f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`.

If you previously relied on turbo cache types for VRAM-constrained models, you must either:
1. Reduce context size instead
2. Increase `-fitt` value
3. Accept higher VRAM usage with q8_0 cache

**Performance comparison (b9743 vs old b9341 mtp-turbo fork):**
| Metric | Old (b9341 fork) | New (b9743 upstream) | Delta |
|--------|-----------------|---------------------|-------|
| gemma-12b t/s | 117.8 | 96.2 | -18% (known regression) |
| qwen36-35b t/s | 45.8 | 51.4 | +12% |
| gemma4-v2 t/s | 54.0 | 52.7 | ~same |
| gemma4-v2 VRAM (ngl 99) | OOM | 7.7 GB fits all layers | Huge improvement |
| Cold start (gemma4-v2) | 25s | 3s | Much faster |
| Eagle3 support | ❌ | ✅ | New capability |

## Provider Cleanup: Free-Tier Only

When a user asks to show only free models from a provider:

### OpenRouter / OpenCode
1. Fetch the model list from the API (Unsloth HF API or OpenRouter models page)
2. Identify free models — they have `-free` suffix on OpenCode Zen, or `:free` suffix on OpenRouter, or `$0/M` pricing
3. **Strip the provider's model list** to only free entries:
   ```python
   cfg['providers'][name]['models'] = {k: v for k, v in models.items() if k.endswith('-free')}
   ```
4. For providers with **no free tier** (e.g., OpenCode Go), **remove the entire provider block**
5. Restart the gateway

### Known Free Models
| Provider | Free models |
|----------|-------------|
| OpenCode Zen | `deepseek-v4-flash-free`, `qwen3.6-plus-free`, `minimax-m3-free`, `mimo-v2.5-free`, `nemotron-3-ultra-free`, `north-mini-code-free` |
| OpenRouter | `nvidia/nemotron-3-ultra-550b-a55b:free`, `nvidia/nemotron-3-super-120b-a12b:free`, `poolside/laguna-m-1:free`, `cohere/north-mini-code:free`, `openai/gpt-oss-120b:free`, `google/gemma-4-31b-it:free` and 6 more |
| NVIDIA NIM | All 121 models free for development (rate-limited) |

### Pitfall: `model_id` format
- OpenRouter uses colon-separated format: `nvidia/nemotron-3-ultra-550b-a55b:free`
- OpenCode uses dash-suffix: `nemotron-3-ultra-free`
- The `:free` suffix is part of the model ID for OpenRouter routing

## Eagle3 Speculator Setup

Eagle3 speculative decoding requires a **speculator head** (small model, ~1 GB) paired with a **matching base model**. The speculator is architecture-locked — it will NOT work with any other base model.

### Workflow

1. **Download the speculator** (safetensors format, ~327 MB to 1 GB):
   ```bash
   mkdir -p /models/downloads/eagle3-speculator
   curl -L -o /models/downloads/eagle3-speculator/model.safetensors "<url>"
   curl -sL -o /models/downloads/eagle3-speculator/config.json "<url>"
   ```

2. **Download the matching base model** as GGUF (same architecture, same size):
   ```bash
   # Find matching GGUF from unsloth or bartowski
   curl -s "https://huggingface.co/api/models?search=<MODEL-NAME>-GGUF&sort=downloads" | python3 -c "import sys,json; [print(m['modelId']) for m in json.load(sys.stdin)[:5]]"
   ```

3. **Convert the speculator to GGUF** (requires the base model's HF config for vocabulary):
   ```bash
   # First download the base model's config files
   pip install huggingface_hub -q
   python3 -c "
   from huggingface_hub import snapshot_download
   snapshot_download('<base-model-org>/<base-model-name>',
       local_dir='/tmp/base-model-config',
       allow_patterns=['tokenizer.json', 'tokenizer_config.json', 'config.json'])
   "

   # Then convert
   python3 convert_hf_to_gguf.py \
     /models/downloads/eagle3-speculator \
     --outfile /models/downloads/eagle3-speculator/eagle3.gguf \
     --target-model-dir /tmp/base-model-config
   ```
   - **--target-model-dir** (NOT `--target-model`) — points to the base model's HF config directory, NOT the base model GGUF file
   - The converter needs the base model's config.json, tokenizer.json, and tokenizer_config.json for vocabulary metadata
   - No `--speculator` flag exists — the converter auto-detects Eagle3 from the model's config.json

### Constraints
- Most Eagle3 speculators are in **safetensors format** (NOT pre-built GGUFs) and require conversion
- Exception: `jdluzen/Qwen3-Coder-Next-Eagle3-GGUF` provides pre-built GGUFs, but they require a llama.cpp build that supports `qwen3next` Eagle3 architecture — NOT yet in mainline as of b9743 (the model's repo explicitly says \"not available in mainline llama.cpp yet\")
- The base model MUST match exactly (same architecture, same parameter count) — the speculator is architecture-locked
- The speculator is tiny (~1 GB at Q4_K_M or ~327 MB bf16) and loads quickly, runs entirely on GPU
- Expected speedup on benchmark hardware (8× GPU, tensor parallel): ~50% over non-speculative inference
- **On single GPU with CPU expert offload (`--n-cpu-moe`), Eagle3 gives marginal improvement** (~1-3 t/s) because the bottleneck is PCIe bandwidth for expert weights, not token prediction speed
   --spec-draft-model /models/downloads/eagle3-speculator/eagle3-speculator.gguf\n   ```\n\n## Multi-Part GGUF Files (Unsloth Dynamic)

Some models (especially UD quants) are released as **split GGUFs**:
```
UD-Q3_K_M/Qwen3.5-122B-A10B-UD-Q3_K_M-00001-of-00003.gguf
UD-Q3_K_M/Qwen3.5-122B-A10B-UD-Q3_K_M-00002-of-00003.gguf
UD-Q3_K_M/Qwen3.5-122B-A10B-UD-Q3_K_M-00003-of-00003.gguf
```

### Handling
- **Download all parts** into the same directory with the original names
- **Do NOT rename the files** — llama.cpp finds continuation files by scanning the directory for the split naming pattern
- **Point llama-server to the FIRST file only** (`...00001-of-00003.gguf`) — it auto-discovers the rest
- **Verification:** `llama-server --model ...00001-of-00003.gguf --no-mmap -c 512` will load without "tensor data out of bounds" errors

### Pitfall: Partial Downloads
If a multi-part download is interrupted, parts may have mismatched sizes. Check:
```bash
# All parts should be within 1% of each other in size (for uniform splits)
ls -lh /path/to/*0000*-of-*.gguf
```
A part that's significantly smaller than others is incomplete.

### CDN Speed Differences
Download speed varies dramatically by HuggingFace account:
- **unsloth**: ~75-100 MB/s (fastest)
- **bartowski, mradermacher**: ~50-80 MB/s
- **lovedheart** (smaller orgs): ~9 MB/s — very slow
- **Qwen official**: ~40-70 MB/s

For slow downloads: use `notify_on_complete=true` + background with 7200s timeout. Check progress with `ls -lh` periodically. Do NOT kill and restart — curl's `-L -o` supports resume if the same filename is used.

**Known overlap:** `new-model-onboarding` skill exists but targets the old hardware setup (GTX 1080 Ti, systemd units, models.py script). It is **outdated** — this `local-model-fleet-management` skill is the current source of truth for the llama-swap based fleet on RTX 2080 Ti. The old skill references GTX 1080 Ti VRAM budgets, systemd-per-model service units, `models.py` scripts, and `hermes-config.path` watchers — none of which exist in the current llama-swap setup. It should be considered deprecated in favor of this skill.

## Pitfall: "Upstream Command Exited Prematurely" — VRAM Contention

When llama-swap reports `"upstream command exited prematurely"`, the root cause is often **not** a bad model file or wrong flags, but **leftover llama-server processes holding VRAM**.

### Debug Flow

1. **Check `nvidia-smi` first** — if free VRAM is <200 MiB, there's contention
2. **Check for stale llama-server processes:**
   ```bash
   pgrep -f "llama-server" | wc -l
   ```
3. **Kill all and retry:**
   ```bash
   pkill -9 -f llama-server
   sleep 3
   nvidia-smi --query-gpu=memory.free --format=csv,noheader  # should show 11002 MiB
   ```
4. **Then retry the model load**

### Example
This session: Mistral Nemo 12B failed to start with `-ngl 1`, `-c 256`, `--no-mmap` — everything minimal. Root cause: multiple test servers from prior benchmarks had consumed all 11GB. After `pkill -9 -f llama-server`, model loaded fine with `-ngl 99` at 80K context.

### Prevention
- Kill test servers explicitly after benchmarking (`pkill -f "<port>"`) before loading a different model
- Use `--no-host` to reduce per-server overhead
- Before debugging model flags, ALWAYS verify VRAM is free

When using `--n-cpu-moe`, the auto-fit tensor override **disables** automatic compute buffer sizing for the overridden layers. This means the model can appear to fit (weights + KV cache within VRAM) but still OOM on compute buffer allocation.

**Symptoms:**
```
E ggml_backend_cuda_buffer_type_alloc_buffer: allocating <N> MiB on device 0: cudaMalloc failed: out of memory
E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size <N>
E llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
```

**Root cause:** The `-fitt` value determines how much VRAM headroom to leave. With `--n-cpu-moe`, the auto-fit skips the overridden layers, so the compute buffer sizing is less accurate. A `-fitt` of 512 leaves only ~500 MB for compute buffers — fine without `--n-cpu-moe`, but insufficient when the override reduces headroom.

**Fix:** Increase `-fitt` to leave more headroom for compute buffers:
- `-fitt 512` → `-fitt 1024` (adds ~500 MB headroom)
- Or reduce `-c` to shrink the KV cache (e.g., 250K → 200K frees ~300 MB)

**Real example (qwen36-35b-mtp, RTX 2080 Ti 11 GB):**
| Context | --n-cpu-moe | -fitt | VRAM | Result |
|---------|-------------|-------|------|--------|
| 250K | 32 | 512 | ~9.2 GB | Compute buffer OOM (needs 781 MB) |
| 250K | 32 | 768 | ~9.2 GB | Still OOM |
| 200K | 32 | 1024 | ~9.2 GB | ✅ Works (compute buffer fits in ~1 GB headroom) |

The KV cache freed by reducing context (250K→200K) plus the increased `-fitt` together provide enough headroom. Either change alone may not suffice.

## Open WebUI Model List Management

Open WebUI stores model IDs in **three separate locations inside the same config JSON** (see `references/openwebui-sqlite-model-list.md`):

1. **`config` table → `data` JSON → `openai.api_configs["0"].model_ids`** — Controls the UI model picker dropdown. This is what the user sees.
2. **`config` table → `data` JSON → `openai.model_ids`** — A separate redundant list that also affects model visibility. This is EASILY FORGOTTEN because it's buried several keys deeper than `api_configs`.
3. **`model` table** — Per-user model registry (determines UI visibility, includes user-specific model entries added through the UI).

**ALL THREE must be updated** when models are added to or removed from llama-swap. The `model` table is the primary gate for UI visibility, but `openai.model_ids` operates as a secondary filter that can silently hide models even when `api_configs` and `model` table are correct.

**Common failure mode:** `api_configs.model_ids` shows the right models in the DB but `openai.model_ids` still has stale entries. The model picker shows old models and hides new ones. Fix both.

**⚠️ If Open WebUI crashes on startup with `TypeError: fromisoformat: argument must be str`**, see `references/openwebui-sqlite-datetime-crash.md` — the `config` table's `updated_at` column has an integer instead of a datetime string.

**Known Open WebUI SQLite corruption: `updated_at` as integer** — Open WebUI can write an integer Unix timestamp to the `config.updated_at` column (e.g., `1782594088`) instead of a datetime string (`2026-06-04 19:32:30`). SQLAlchemy's `str_to_datetime` processor calls `.fromisoformat()` on the value, which crashes with `TypeError: fromisoformat: argument must be str`. This prevents Open WebUI from starting (restart loop). Fix:
```sql
sqlite3 /path/to/webui.db
UPDATE config SET updated_at = datetime(updated_at, 'unixepoch') WHERE typeof(updated_at) = 'integer';
```

### Quick Add (Manual)
```bash
docker exec open-webui python3 -c "
import sqlite3, time
conn = sqlite3.connect('/app/backend/data/webui.db')
uid = conn.execute(\"SELECT id FROM user WHERE role='admin' LIMIT 1\").fetchone()[0]
now = int(time.time())
for mid in ['<model-id-1>', '<model-id-2>']:
    if not conn.execute('SELECT id FROM model WHERE base_model_id=?',(mid,)).fetchone():
        conn.execute('INSERT INTO model VALUES (?,?,?,?,?,?,?,?,1)', (f'model-{mid}', uid, mid, mid, '{}', '{}', now, now))
        print(f'Added: {mid}')
conn.commit()
"
docker restart open-webui
```

### Bulk Sync Script
See `references/openwebui-sqlite-model-list.md` → "Bulk Sync" section for a script that matches the Open WebUI model list to llama-swap's current fleet exactly.

### Config Location
Open WebUI runs as a Docker container (ghcr.io/open-webui/open-webui:latest, port 3081), with DB at `/app/backend/data/webui.db`.

## CDN Speed Differences (HF downloads)
Download speed varies dramatically by HuggingFace account:
- **unsloth**: ~75-100 MB/s (fastest)
- **bartowski, mradermacher**: ~50-80 MB/s
- **lovedheart** (smaller orgs): ~9 MB/s — very slow
- **Qwen official**: ~40-70 MB/s

For slow downloads: use `notify_on_complete=true` + background with 7200s timeout using the `hf` CLI (replaces deprecated `huggingface-cli`). Check progress with `ls -lh` periodically. Do NOT kill and restart — `hf download` supports resume.

### DiffusionGemma (PR #24423)
- Architecture: `diffusion_gemma` — NOT supported by `llama-server` (HTTP)
- Only has `llama-diffusion-cli` (CLI) and `llama-diffusion-gemma-server` (stdin/stdout protocol)
- Config uses **env vars** not CLI flags: `NGL` (GPU layers), `MAXTOK` (context), `FA` (flash attention)
- `--n-cpu-moe 128` works with the CLI for MoE offload, but NOT with the server
- Q4_K_M = 16 GB file, needs 18 GB total memory — runs at ~8 tok/s with offload on 11 GB VRAM
- To build: checkout PR #24423, build `llama-diffusion-cli` target
- When nvcc fails with g++-14: use `-DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-13`
- The PR is still a Draft — no HTTP server support yet

### Forced Context Extension
When a model's native context is too small (e.g., Moonlight at 8K), forced extension is possible but degrades quality:
```bash
# Required for models without YaRN scaling in GGUF metadata
--override-kv <arch>.context_length=int:<desired_ctx>
--rope-scaling linear --rope-freq-scale <factor>
```
Example for Moonlight (deepseek2, 8K native → 32K):
```
--override-kv deepseek2.context_length=int:32768
--rope-scaling linear --rope-freq-scale 4.0
```
**WARNING:** Linear scaling at 4x on a model not trained for extended context produces garbage output (token repetition). Only use if quality is not critical or the model documentation explicitly supports it.

The `context_length` in Hermes `custom_providers` must match the actual `-c` value in llama-swap config for every model. Mismatch causes silent context truncation: Hermes thinks the model supports 256K but llama-swap only allocated 131K.

**How to audit:**

```bash
# 1. Extract -c values from llama-swap config
grep -E '^\s+-c [0-9]+' ~/.config/llama-swap/config.yaml | while read line; do
  echo "$line"
done

# 2. Extract context_length values from Hermes config
python3 -c "
import yaml
with open('/home/rurouni/.hermes/config.yaml') as f:
    c = yaml.safe_load(f)
for m in c['custom_providers'][0]['models']:
    print(f'{m[\"id\"]}: {m[\"context_length\"]}')
"

# 3. Cross-reference: each model's -c must equal its context_length
# Verify the model IDs match too (llama-swap model key == custom_provider model id)
```

**Common mismatches found in the wild:**
- `gemma-12b`: llama-swap uses `-c 131072`, config had `256000` — double the actual capacity
- `gemma-26b`: llama-swap uses `-c 200000`, config had `256000` — overshoots VRAM headroom
- `deepseek-v2-lite`: entirely missing from Hermes config despite being in llama-swap

**Fix pattern when `hermes config set` fails** (stores arrays as JSON strings):
```python
import yaml, os
path = os.path.expanduser("~/.hermes/config.yaml")
with open(path) as f:
    data = yaml.safe_load(f)
# Fix models array (index 0 for first custom_provider)
data["custom_providers"][0]["models"] = <correct-list>
with open(path, "w") as f:
    yaml.dump(data, f, default_flow_style=False, sort_keys=False, allow_unicode=True, width=120)
```

```bash
# Verify the update didn't corrupt the YAML
python3 -c "import yaml; yaml.safe_load(open('/home/rurouni/.hermes/config.yaml'))" && echo "Valid YAML"
```

**⚠️ CRITICAL: When writing Hermes config via Python, use EXACTLY these kwargs:**
```python
yaml.dump(data, f, default_flow_style=False, sort_keys=False, width=1000)
```
A typo like `default_flow_state` instead of `default_flow_style` will **silently write an empty file**, destroying the config. Always validate the YAML after writing. Keep a recent backup accessible for quick restore:
```bash
# Restore from latest backup
cp ~/.hermes/config.yaml.bak-* ~/.hermes/config.yaml
```

### Hermes Config Corruption Recovery

If the config file is emptied (e.g., by a failed `yaml.dump()` with wrong kwargs):

1. **Check if the file is empty:** `wc -c ~/.hermes/config.yaml`
2. **Restore from latest backup:**
   ```bash
   ls -la ~/.hermes/config.yaml.bak-*  # find most recent
   cp ~/.hermes/config.yaml.bak-20260619-223108 ~/.hermes/config.yaml
   ```
3. **Or restore from a state snapshot:**
   ```bash
   ls -la ~/.hermes/state-snapshots/*/config.yaml
   cp ~/.hermes/state-snapshots/latest/config.yaml ~/.hermes/config.yaml
   ```
4. **Re-apply changes that were lost** (new models, provider edits) — use `python3 << 'PYEOF'` with the correct kwargs
5. **Validate after write:**
   ```bash
   python3 -c "import yaml; yaml.safe_load(open('/home/rurouni/.hermes/config.yaml'))" && echo "Valid YAML"
   ```
6. **Restart gateway:** `kill $(pgrep -f 'hermes.*gateway.run')` then restart via background
7. **Always verify gateway is responding:** `curl -s :18789/v1/models` should return `{"error":{"message":"Invalid API key"...}}` (key-required = alive) not `Connection refused` (dead)

**Pitfall:** After restoring from a snapshot, model IDs may be stale (e.g., `gemma-26b` vs `gemma-26b-200k`). Check and fix any drift between llama-swap model keys and Hermes model IDs.

## P0 Flag Verification Checklist

After any config change to all models, run:

```bash
grep -c "chat-template-kwargs" ~/.config/llama-swap/config.yaml     # must be 0
grep -c "reasoning off" ~/.config/llama-swap/config.yaml             # must equal model count (except Qwen — see pitfall)
grep -c "spec-draft-type-k" ~/.config/llama-swap/config.yaml         # must cover all MTP/draft-mtp models
grep "min-p 0 " ~/.config/llama-swap/config.yaml                     # must be empty (min-p=0 is broken)
grep -c "no-host -fitt" ~/.config/llama-swap/config.yaml             # should match model count
grep "mlock\\|no-mmap" ~/.config/llama-swap/config.yaml | head -3     # --no-mmap requires --mlock
```

**Pitfall: `reasoning_content` field vs `content` field.** GLM-4.7-flash-reap natively outputs to `reasoning_content` instead of `content`, leaving `content` empty. This is NOT caught by any of the above checks. To verify GLM works:

```bash
# Test that GLM produces non-empty content
curl -s :9292/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"glm-4.7-flash-reap","messages":[{"role":"user","content":"say hi"}],"max_tokens":20}' | \
  python3 -c "import sys,json; m=json.load(sys.stdin)['choices'][0]['message']; print('content:', m.get('content','')); print('reasoning:', m.get('reasoning_content',''))"
# content must not be empty — if empty, add --reasoning off to the model's cmd
```

**Known fact:** `--reasoning off` does NOT suppress Qwen models' native `<think>` tags. The check above will show Qwen models still emitting thinking tags despite the flag. This is expected — the flag prevents the server from ADDITIONALLY injecting reasoning but cannot strip model-native output. Open WebUI's Thinking toggle is the only way to suppress display.

## Model Thinking/Reasoning Behavior Reference

Some models natively output reasoning/thinking content regardless of `--reasoning off`. This affects how they behave in Open WebUI, API clients, and chat history.

### Which Models Produce Native Thinking

| Model | Arch | Native Thinking | Manifestation | `--reasoning off` works? |
|---|---|---|---|---|
| **gemma-26b-200k** | Gemma 4 MoE | No | Clean output | N/A |
| **glm-4.7-flash** | GLM MoE | **Yes** | Outputs `reasoning_content` field, `content` is empty | ✅ Fully — REQUIRED |
| **qwen36-35b-mtp** | Qwen 3.6 MoE | **Yes** | Native `<think>`/`</think>` tags in response | ❌ Tags persist |
| **ornith-35b-q6-mtp** (APEX) | Qwen3.5 MoE | Yes | `<think>` tags in response | Works — 262K context, 28.5 t/s warm |
| **ornith-9b-mtp-q5** | Qwen3.5 Dense MTP | **Yes** | `<think>` tags in response | Works — keep enabled; 71.3 t/s |
| **qwythos-9b-mtp-q6** | Qwen3.5 Dense | Yes | `<think>` tags in response (Claude Mythos trace format) | Works |

### ⚠️ Dense Reasoning Models: KV Cache OOM Masquerading as Tag Issue

**Initial suspicion:** `--reasoning off` on dense models (Ornith-9B) causes intermittent empty responses in Open WebUI.
**Actual root cause after investigation:** **KV cache OOM.** At `-c 131072` without `--no-kv-offload`, the KV cache allocates ~4.6 GB (36 layers × 4 KV heads × 128 hidden × 2 × 128K tokens × q8_0 byte). Combined with 6.9 GB weights = ~11.5 GB → exceeds 11 GB RTX 2080 Ti. The server crashes mid-stream (upstream disconnection), leaving Open WebUI with empty content.

**Evidence:**
- Streaming with `--reasoning off`: 0 reasoning events, 257 content events, clean completion
- Streaming without `--reasoning off`: 185 reasoning events + 9 content events per "hello" — excessive thinking tokens
- Direct API calls work fine; only Docker-streaming via Open WebUI shows the crash (because server dies mid-stream)
- llama-swap logs show `recovered from upstream disconnection during streaming` + `no valid JSON data found in stream`

**Fix (NOT removing `--reasoning off`):**
1. **Reduce context** to 32768 (frees ~3.5 GB KV cache → fits in 11 GB)
2. **Add `--no-kv-offload`** (move KV cache to system RAM)
3. **Keep `--reasoning off`** — works fine on dense models; empty responses were OOM crashes, not tag stripping
4. **Do NOT remove `--reasoning off`** — without it, model outputs 185+ thinking tokens per request, filling KV cache and slowing responses

**For MoE models** (ornith-35b, qwen36-35b): Same KV cache risk applies at high context. `--no-kv-offload` is always recommended for 11 GB cards with >64K context.

### Testing a Model for Native Thinking

```bash
curl -s :9292/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"<model-id>","messages":[{"role":"user","content":"say hi"}],"max_tokens":50}' | \
  python3 -c "
import sys,json
d=json.load(sys.stdin)
m=d['choices'][0]['message']
print('content:', repr(m.get('content','')))
print('reasoning:', repr(m.get('reasoning_content','')))
"
```
- If `reasoning_content` is non-empty → model has native thinking in `reasoning_content` field
- If `content` contains `<think>` or `<thinking>` tags → Qwen-style native thinking
- If both are clean → no native thinking

### Open WebUI Interaction

See `references/openwebui-thinking-toggle.md` for the critical interaction between Open WebUI's Thinking toggle and model routing. In short: **turning the toggle ON can cause Open WebUI to auto-select a reasoning-capable model** even when the user has a different model selected in the dropdown.

### GLM-4.7-flash-reap Specifics

- **Must** have `--reasoning off` — without it, the model outputs ONLY `reasoning_content` (empty `content` = invisible response)
- OOM fix at 131K q8_0 KV cache on 11 GB: increase `--n-cpu-moe` from 20 to 30 (frees ~0.8 GB, -20% t/s)
- Alternative: drop KV to q4_0 (keeps speed, small quality loss)
- Benchmark: 42.7 t/s at q8_0/q8_0 with n-cpu-moe 30; 53.5 t/s at q4_0/q4_0 with n-cpu-moe 20

Full detail in `references/` (22 files covering vram-table, hf-model-research, system-tuning, service-lifecycle, routing, cost-analysis, openwebui-thinking-toggle, and more). Latest: `references/2026-07-03-fleet-overhaul-apex-mtp.md` — Jul 3 fleet overhaul with APEX upgrade, Ornith-9B-MTP replacement, Qwythos v3, GLM removal, repeat-penalty flags.