# Fleet Stress Test & Audit Methodology

## When to Use

The user asks to "stress test", "audit", "bench test", or "scorecard" the fleet; or wants to decide what to keep/remove/replace.

## Full Stress-Test Pipeline

### Phase 1: Research (per model)

For each model, gather data across 5 dimensions:

1. **Architecture** — model card (HF `config.json`), GGUF metadata (`scripts/gguf-metadata-scan.py`): params, layers, KV heads, head dim, MoE count, native context
2. **Community validation** — Reddit (r/LocalLLaMA), HuggingFace downloads/mo, independent benchmarks, known issues
3. **Optimal flags** — model card recommended sampling, Unsloth docs, community llama.cpp configs
4. **Hardware benchmarks** — t/s on comparable GPUs (RTX 3060 12 GB, RTX 4070 12 GB, RTX 4090)
5. **Unique characteristics** — MTP support, native thinking, abliteration, architecture quirks, special requirements

Search queries template:
```
web_search(query="<model-name> llama.cpp benchmarks flags", limit=5)
web_search(query="<model-name> model review community", limit=5)
web_extract(urls=["https://huggingface.co/<org>/<model>"])
```

### Phase 2: Current State Audit

Check three-way alignment: **disk ↔ llama-swap config ↔ Hermes custom_providers**

```bash
# Models on disk
ls -lh /models/downloads/*.gguf /models/downloads/*/*.gguf 2>/dev/null

# Models in llama-swap
curl -s http://127.0.0.1:9292/v1/models

# Models in Hermes
grep -A3 'id:' ~/.hermes/config.yaml | grep -E 'id:|context_length:'
```

Also check:
- Disk orphans (files on disk not referenced in any config)
- Stale config references (models in config with no file)
- Hermes `context_length` mismatches vs llama-swap `-c` values
- AGENTS.md mentions of fleet (often stale)

### Phase 3: Redundancy Assessment

Score each model against these criteria:

| Criterion | Question | Action |
|-----------|----------|--------|
| Role overlap | Does another model serve the same role? | Drop the weaker one |
| Speed ratio | >50% slower than the class leader? | Drop if no unique capability |
| Community validation | <100 HF downloads/mo or <1 month old? | Prefer well-documented alternatives |
| VRAM safety | Uses >90% of 11 GB? | Reduce ctx/quant or drop |
| Disk cost | >20 GB for a niche model? | Justify or drop |
| Actual usage | Used in the last 30 days? | Drop if not |

### Phase 4: Benchmark

Use the v2 benchmark script pattern through llama-swap (each model cold-loads automatically):

```python
# Key measurements per model:
# - Cold load time (first request, includes model switch)
# - Generation speed (short prompt, 500 tok output, 2 runs)
# - Prompt processing speed (long prompt, 5 tok output)
# - VRAM usage (nvidia-smi after model load)
```

See `scripts/bench-fleet-v2.py` for the reusable benchmark script.

### Phase 5: Scorecard & Decision

Build a tiered scorecard:

| Tier | Meaning | Example |
|------|---------|---------|
| 🏆 Keep | Elite — best-in-class for a real role | Primary daily driver, deep reasoning, fastest |
| 🟡 Conditional | Useful but overlapping | Could live without, nice to have |
| 🔴 Remove | Redundant/niche/dead weight | Replaced by better model, never used |

For each removal, track: disk freed, VRAM freed, replacement model.

### Phase 6: Cross-System Cleanup

9-step removal procedure (see SKILL.md → "Full Fleet Cleanup Procedure (Cross-System Sync)"):

1. Delete GGUF files from disk
2. Remove from llama-swap config
3. Remove from Hermes custom_providers
4. Remove from Hermes fallback_providers
5. Remove from OpenCode local-gateway config
6. Remove from Open WebUI SQLite (model table + config JSON — both `api_configs.0.model_ids` AND `openai.model_ids`)
7. Update AGENTS.md
8. Update memory fleet entry
9. Restart llama-swap

**Context cross-reference after cleanup** — verify every system matches:
```bash
python3 << 'PYEOF'
import yaml, json, sqlite3, re, subprocess

# 1. llama-swap API
r = subprocess.run(["curl", "-s", "http://127.0.0.1:9292/v1/models"], capture_output=True, text=True)
swap = {m["id"]: m.get("context_length","?") for m in json.loads(r.stdout)["data"]}

# 2. Hermes config
with open("/home/rurouni/.hermes/config.yaml") as f:
    h = yaml.safe_load(f)
hermes = {}
for cp in h.get("custom_providers", []):
    for m in cp.get("models", []):
        hermes[m["id"]] = m.get("context_length","?")

# 3. OpenCode config
with open("/home/rurouni/.config/opencode/opencode.json") as f:
    oc = json.load(f)
ocode = {mid: m["limit"]["context"] for mid, m in oc["provider"]["local-gateway"]["models"].items()}

# 4. Open WebUI
conn = sqlite3.connect("/home/rurouni/openwebui-data/webui.db")
owu_cfg = json.loads(conn.execute("SELECT data FROM config LIMIT 1").fetchone()[0])
owu_api = set(owu_cfg["openai"]["api_configs"]["0"]["model_ids"])
owu_meta = set(owu_cfg["openai"]["model_ids"])
owu_model_tbl = {r[0] for r in conn.execute("SELECT base_model_id FROM model").fetchall()}
conn.close()

print(f"{'Model':<30} {'Swap':>8} {'Hermes':>8} {'OpenCode':>8} {'OWU_tbl':>8} {'OWU_api':>8} {'OWU_meta':>8}")
for mid in sorted(swap):
    print(f"{mid:<30} {swap.get(mid,'?'):>8} {hermes.get(mid,'?'):>8} {ocode.get(mid,'?'):>8} {'Y' if mid in owu_model_tbl else '-':>8} {'Y' if mid in owu_api else '-':>8} {'Y' if mid in owu_meta else '-':>8}")
PYEOF
```

### Phase 7: Tweak Audit

After cleanup, compare each remaining model's flags against model-card recommendations:

| What to check | Where to find it |
|---------------|-----------------|
| Temperature / top-p / top-k | HF model card README, Unsloth docs |
| --reasoning off requirement | Know which models need it (GLM) vs where it's optional |
| MTP draft-n-max | Community benchmarks on comparable hardware |
| VRAM safety margin | <500 MiB free → add --no-warmup or lower -fitt |
| Sampling params deviating from recommended | E.g., Gemma at temp=0.2 instead of temp=1.0 |

## Quant Upgrade Trade-off Analysis

When considering whether a lower quantization (e.g., Q6→Q5) can fund a context increase:

### 1. Find file sizes for both quants
```
# Check HF repo for quant file sizes
Search: <model-name>-GGUF on huggingface.co
```

### 2. Calculate VRAM delta
- **Dense model**: ∆VRAM ≈ ∆file_size (since -ngl 99 puts everything on GPU)
- **MoE model with --n-cpu-moe**: ∆VRAM ≈ ∆non_expert_weights + ∆(ncm layers × per_layer_cost)
  - Non-expert weights ≈ 30-35% of total file
  - Each expert layer at Q6/Q5: depends on architecture

### 3. Calculate KV cache cost at target context
```
KV per token (q8_0) = 2 × n_layers × n_kv_heads × head_dim bytes
Total KV = KV_per_token × context_size
```

### 4. Budget check
```
Total VRAM = model_weights_on_GPU + KV_cache + 0.5 GB overhead
Must be < 10,500 MiB for stability on 11 GB card
```

### 5. Context ceiling formula
```
max_ctx = (11000 - model_weights - 500) / KV_per_token
```
Where 500 MiB is the minimum safe headroom for compute buffers and OS.

### Real example (Ornith-9B, RTX 2080 Ti 11 GB):
| Quant | File | GPU Weights | KV/64K | Total | Feasible ctx |
|-------|------|-------------|--------|-------|-------------|
| Q6_K | 6.9 GB | 6.9 GB | 2.3 GB | 9.2 GB | 64K (no --no-kv-offload) |
| Q5_K_M | 6.47 GB | 6.5 GB | 2.3 GB | 8.8 GB | 64K ✅ (0.4 GB saved) |
| Q5_K_M | 6.47 GB | 6.5 GB | 3.5 GB | 10.0 GB | 96K ✅ stretch |