# Local Model Fleet Management — GGUF Notes

## Path Audit After Model Reorg

After any model directory reorg, llama-swap config silently breaks. All models will list in `/v1/models` but fail on first request with opaque errors.

```bash
grep -oP -- '-m \S+' ~/.config/llama-swap/config.yaml | while read flag path; do
  [ -f "$path" ] && echo "✓ $path" || echo "✗ MISSING: $path"
done
```

## Three-Way Alignment Audit

Run `scripts/cross-ref-audit.py` to verify disk ↔ llama-swap ↔ Hermes:

```bash
# Source 1: Models on disk
echo "=== DISK ==="
find /models/downloads/ -name '*.gguf' -maxdepth 2 | sort

# Source 2: llama-swap config
echo "=== LLAMA-SWAP ==="
grep -oP -- '-m \S+' ~/.config/llama-swap/config.yaml | while read flag path; do
  [ -f "$path" ] && echo "✓ $path" || echo "✗ MISSING: $path"
done

# Source 3: Hermes custom_providers
echo "=== HERMES ==="
python3 -c "
import yaml
c = yaml.safe_load(open('$HOME/.hermes/config.yaml'))
for p in c.get('custom_providers', []):
    for m in p.get('models', []):
        print(f'  {m[\"id\"]} → {p[\"base_url\"]}')
"
```

## Migration Checklist (GPU Upgrade)

1. Verify GPU detection: `nvidia-smi`, `lspci | grep -i nvidia`, compute capability
2. Raise power limit: `sudo nvidia-smi -pl <max>`
3. Enable persistence mode: `sudo nvidia-smi -pm 1`
4. System tuning: THP → madvise, nofile → 65536, memlock=infinity
5. Verify CUDA binary matches GPU SM version
6. Audit all model paths (llama-swap config, systemd units, Hermes custom_providers)
7. Re-benchmark every model

## Benchmark Test Matrix

| Variable | Values to test |
|----------|---------------|
| Cache type | turbo3/turbo4, q8_0/q8_0 |
| Context | 128K, 150K, 200K, 250K |
| Threads | 16, 32 (match physical cores) |
| MTP | on, off (for models that support it) |

## Turing-Specific Flags (SM75)

```
--flash-attn on          # Tensor core accelerated attention
--cache-type-k turbo3    # INT8-quantized K cache
--cache-type-v turbo4    # INT4-quantized V cache
--no-host -fitt 512      # Tighter VRAM budget, frees ~500MB
--threads 16             # 16 for 35B MoE (faster than 32)
--threads 32             # 32 for non-MoE 9B models
```

**Exception for MTP stability:** When MTP hangs occur, replace with `--cache-type-k q8_0 --cache-type-v q8_0`.

## Gemma 26B is MoE
Use `--n-cpu-moe 128` (or up to 256) just like Qwen MoE models. The `q8_0` cache fix also applies for flash-attn + MTP stability on Turing.