# Local llama.cpp Model Diagnostics

Use this when a local GGUF model is slow, hallucinates on tiny prompts, repeats itself, or Hermes reports connection errors after `/model` switches.

## Fast triage

1. Check Hermes config and custom provider mapping:
   ```bash
   python3 - <<'PY'
   import yaml, json
   cfg=yaml.safe_load(open('/home/rurouni/.hermes/config.yaml')) or {}
   print(json.dumps(cfg.get('model', {}), indent=2))
   for cp in cfg.get('custom_providers', []) or []:
       if cp.get('name') == 'local':
           print(json.dumps(cp.get('models', {}), indent=2)[:4000])
   PY
   ```
2. Check services, ports, and VRAM:
   ```bash
   systemctl --no-pager --plain list-units 'llama-server*.service' --state=active
   ss -ltnp | grep -E ':808[0-9]|:8090' || true
   nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv,noheader,nounits
   ```
3. Check gateway errors for wrong-port calls:
   ```bash
   journalctl --user -u hermes-gateway.service --since '30 min ago' --no-pager -o short-iso \
     | grep -E 'APIConnectionError|base_url=http://127.0.0.1|model=' | tail -80
   ```
4. Check model switching races/duplicates:
   ```bash
   tail -120 ~/.hermes/logs/model-switch.log
   systemctl --user is-active hermes-config.path || true
   systemctl is-active hermes-config.path || true
   ```

## Direct probe pattern

Probe llama-server directly before blaming Hermes. This bypasses gateway session overrides and provider resolution.

```bash
python3 - <<'PY'
import json, time, urllib.request
port=8088
model='phi-4-Q4_K.gguf'
payload={
  'model': model,
  'messages': [{'role':'user','content':'hi'}],
  'max_tokens': 80,
  'temperature': 0.2,
  'stream': False,
}
req=urllib.request.Request(
  f'http://127.0.0.1:{port}/v1/chat/completions',
  data=json.dumps(payload).encode(),
  headers={'Content-Type':'application/json'},
)
t=time.time()
data=json.loads(urllib.request.urlopen(req,timeout=120).read())
print('latency_s', round(time.time()-t, 2))
print(data['choices'][0]['message']['content'])
PY
```

Healthy tiny-prompt responses should be short and relevant. If `hi` produces a long random code-help answer, treat the model/service configuration as suspect.

## Inspect GGUF native context and template

```bash
python3 - <<'PY'
from gguf import GGUFReader
path='/models/llm/phi/phi-4-Q4_K.gguf'
r=GGUFReader(path)
for k in sorted(r.fields):
    if 'context_length' in k or k == 'tokenizer.chat_template' or k in ('general.name','tokenizer.ggml.model','tokenizer.ggml.pre'):
        f=r.fields[k]
        print('\n###', k)
        if f.types and str(f.types[-1]).endswith('STRING: 8>'):
            print(bytes(f.parts[f.data[0]].tolist()).decode('utf-8', errors='replace'))
        else:
            print(f)
PY
```

If the service forces a much larger context than the GGUF advertises, verify behavior at the native context before assuming the model is bad.

## Phi-4 lesson learned

`/models/llm/phi/phi-4-Q4_K.gguf` advertises native context `16384`. Running it as a fake 65K model with `--rope-scale 8.0` caused:

- irrelevant repetitive hallucinations on `hi`
- slower tiny-prompt latency (~3-4s for 80 tokens)
- output that looked like stale prompt bleed, even with a clean direct API call

Fix used:

- set `llama-server-phi4.service` to `-c 16384 --ctx-size 16384`
- remove `--rope-scale 8.0`
- keep `--cache-type-k turbo4 --cache-type-v turbo4` (TurboKV at 16K tested healthy)
- update `custom_providers.local.models.phi-4-Q4_K.gguf.context_length` to `16384`

Verification after fix:

- `hi` -> `Hello! How can I assist you today?`
- direct API latency ~0.5s

## Duplicate watcher pitfall

Do not allow both user and system `hermes-config.path` watchers to fire for the same config file. Duplicate triggers create duplicate log lines and can race service swaps.

Check:
```bash
systemctl --user is-active hermes-config.path || true
systemctl is-active hermes-config.path || true
```

Keep exactly one active watcher for the installation's chosen architecture. In Ray's setup, the user watcher is the intended active watcher; the duplicate system watcher was disabled.

## Style note for live troubleshooting

When the user is frustrated by broken local models, do not answer like customer support or ask generic survey questions. Start with disk-backed diagnostics immediately: config, ports, services, logs, and direct API probes. Report concrete findings and fixes, not reassurance.