# Local Model Connection Diagnosis — Session Debug Notes

From May 12 2026 session: user reported "Local model is still not connecting" after switching from gemma-3-12b-it-qat-Q4_0.gguf to deepseek-v4-flash (cloud via DeepSeek).

## Root Cause

`hermes-config.path` systemd path watcher was `inactive (dead)` since 17:29 that day. It had silently deactivated — no error, no signal — just stopped watching config.yaml for changes.

**Impact:** When the user used `/model` in Telegram to switch to DeepSeek cloud at ~23:06, the gateway correctly wrote `model.default: deepseek-v4-flash` + `model.provider: deepseek` to config.yaml, but with no path watcher listening, `auto-model-switch.py` never fired. The old Gemma-3-12B-QAT llama-server kept running, consuming 11.7GB RAM + 8.2GB VRAM, as an orphan.

## Diagnosis Steps Taken

### Config Check
```
model:
  default: deepseek-v4-flash
  provider: deepseek
  base_url: https://api.deepseek.com/v1
```
Config was correct — cloud model, no local service needed. The model path `/models/llm/deepseek/flash/deepseek-v4-flash.gguf` does not exist on disk (it's a cloud model, not a local GGUF).

### Path Watcher Status
```bash
$ sudo systemctl status hermes-config.path
  hermes-config.path - Watch Hermes config.yaml for model changes
     Loaded: loaded (/etc/systemd/system/hermes-config.path; disabled; preset: enabled)
     Active: inactive (dead)

$ sudo journalctl -u hermes-config.path --no-pager
May 12 14:47:26 Started hermes-config.path
May 12 17:29:31 hermes-config.path: Deactivated successfully.
May 12 17:29:31 Stopped hermes-config.path
```
Last deactivation at 17:29, never restarted. The path unit's service companion:
```bash
$ sudo systemctl status hermes-config.service
  hermes-config.service - Hermes Agent: handle config.yaml model change
     Active: inactive (dead)
```
Last journal activity was at 17:23 (switched to Gemma 3 12B). No further triggers.

### Running Services vs Systemd States
```bash
$ sudo systemctl list-units | grep -i llama-server
llama-server-gemma3-qat.service    loaded active running   Gemma 3 12B QAT
llama-server-smol.service          loaded active running   SmolLM3-3B CPU

$ ps aux | grep llama-server | grep -v grep
PID 552775  llama-server SmolLM3-3B CPU (preserved background)
PID 556357  llama-server gemma-3-12b-it-qat-Q4_0.gguf (orphan! 11.7GB RAM)
```

### System vs User-Level Units
All llama-server services are system-level (`/etc/systemd/system/llama-server*.service`), NOT user-level. Using `systemctl --user` to check them returns empty/no matches. Always use `sudo systemctl` or plain `systemctl` (system-level default).

### Auto-Switch Log Timeline
```
22:48:28 Config changed to local model: Qwen3.6-35B-A3B-UB (already on it)
22:50:24 Config changed to local model: Ministral-3-8B-Reasoning (started OK)
22:55:29 Config changed to local model: Qwen3-8B (started OK)
22:55:41 Config changed to local model: Qwen3-14B (stopped Qwen3-8B, started OK)
22:59:57 Config changed to local model: Gemma-3-12B-QAT (stopped Qwen3-14B, started OK)
23:06:17 Config set to cloud model: deepseek-v4-flash — no service to swap
```
Note: auto-switch was triggered by manual invocations or gateway-side hooks, despite the path watcher being dead. The 23:06 entry shows it DID detect the cloud switch but took no action (correct behavior: don't start a local service for cloud models, but also didn't stop the old one because the code doesn't handle local-to-cloud cleanups).

### DeepSeek API Probe
```bash
$ curl -s https://api.deepseek.com/v1/models -H "Authorization: Bearer $(grep '^DEEPSEEK_API_KEY=' ~/.hermes/.env | cut -d= -f2)"
{"object":"list","data":[{"id":"deepseek-v4-flash","object":"model","owned_by":"deepseek"},{"id":"deepseek-v4-pro","object":"model","owned_by":"deepseek"}]}
```
API was working fine throughout.

## Fix Applied

1. **Restart path watcher:** `sudo systemctl start hermes-config.path`
2. **Stop orphan:** `sudo systemctl stop llama-server-gemma3-qat`
3. **Verify:** VRAM dropped from 8,235 MiB to 641 MiB. Only SmolLM3 CPU remained running.

## Key Lesson

The path watcher is a single point of failure for all model switching. When it's dead, the `/model` picker still writes correct config.yaml changes, but the service-level swap never happens. The symptom is confusing: the user sees the model name change in their session but the old model's GPU process keeps running and the new one never starts.

Always check `systemctl is-active hermes-config.path` as the FIRST step in any "local model not connecting" diagnosis.