---
name: local-model-switch-optimization
title: Local Model Switch Optimization
summary: Optimized techniques for reliable local model switching with VRAM management and systemd resolution
description: Reliable local model switching in systemd environments, focusing on VRAM optimization and service lifecycle management
doc_type: technical guide
---

## Summary
This skill consolidates fixes and best practices for managing local model switches in systemd-managed environments. Key patterns include:
- VRAM contention prevention during local↔local transitions
- Keeping local models warm during cloud model switches
- Eliminating racing conditions in service switching

## Core Techniques
1. **VRAM Management Protocol**
   ```python
   # Only apply between local models
   if active and target:
       time.sleep(3)  # Crucial VRAM free time before restart
   ```
2. **Service Lifecycle Management**
   - Always update config *before* touching services
   - Stop old local models *before* starting new ones
   - Never kill existing services when switching to cloud

3. **Preserved Background Services Pattern**
   CPU-only models (ngl 0) must NEVER be stopped during GPU model switches. Mark them in models.py with `"preserve": True` and have the model-switcher skip them. Currently no CPU-only model is wired into the fleet, but keep this pattern ready if a background service is added:

   ```python
   {
       "name": "Some-CPU-Model",
       "service": "llama-server-some-cpu",
       "preserve": True,
       ...
   }
   ```

   Key insight: `get_active_service()` iterates ALL MODELS, so a background-only CPU service would be returned as "active" and incorrectly stopped without this guard.


4. **Service-Per-Model Unit Files**
   Each model MUST have its own systemd unit file unless that unit is intentionally the dedicated service for a single model. A shared generic unit that gets reused across multiple models is unsafe because its ExecStart hardcodes one GGUF path. Current known-good examples:
   ```
   llama-server.service        → Qwen3.6-35B-A3B-UD (port 8081)
   llama-server-9b.service     → Qwen3.5-9B-UD (port 8082)
   llama-server-gemma4.service → Gemma-4-26B (port 8090)
   ```
   The `models.py` `"service"` field must point to the real backing unit for that exact model.

5. **Model Inventory Must Match Disk Exactly**
   Phantom models in models.py (defined but no .gguf on disk) cause silent failures. Before each session:
   ```bash
   find /models/llm/ -name '*.gguf' | sort
   ```
   Then cross-reference against `models.py` and remove any entry with no matching file. Ports and context sizes must also be verified against the actual systemd unit files.

6. **Four-Way Alignment Audit (Disk ↔ models.py ↔ systemd ↔ custom_providers)**  
   All four sources must agree. Any pair out of sync causes a different failure mode:
   - **Disk ↔ models.py**: missing model = /model selection silently fails
   - **models.py ↔ systemd**: port/service mismatch = wrong service started or switch hangs
   - **models.py ↔ custom_providers**: model present in /model picker but not in models.py → no auto-switch runs
   - **custom_providers ↔ disk**: picker shows dead models; missing models invisible in picker
   
   **Name resolution:** custom_providers model keys must match either `name` or `model_name` in models.py. By default auto-model-switch.py only matches `model_name` (GGUF filename). If you use friendly `name` values as keys, update auto-model-switch.py's lookups and `is_cloud_model()` to match on both fields (see Option D in `references/custom-providers-naming.md`).

   Run the full audit:
   ```bash
   # Source 1: Models on disk
   echo "=== DISK ==="
   find /models/llm/ -name '*.gguf' -maxdepth 4 | sort

   # Source 2: models.py entries
   echo "=== MODELS.PY ==="
   python3 -c "from models import MODELS; [print(m['name'], m['model_name'], m['port'], m['service']) for m in MODELS]" 2>/dev/null \
     || grep -oP '"name": "\K[^"]+' ~/.hermes/scripts/models.py | sort

   # Source 3: systemd services
   echo "=== SYSTEMD ==="
   for f in /etc/systemd/system/llama-server*.service; do
     name=$(basename "$f" .service)
     port=$(grep -oP -- '--port \K\d+' "$f" 2>/dev/null || echo "?")
     model=$(grep -oP -- '-m /\S+/\K[^/]+\.gguf' "$f" 2>/dev/null || echo "?")
     echo "$name: port=$port model=$model"
   done

   # Source 4: custom_providers in config.yaml
   echo "=== CUSTOM_PROVIDERS ==="
   grep -A 1 "models:" ~/.hermes/config.yaml | grep -E "^\s+[^:]" | tr -d ' :' | sort
   ```

   Fix each discrepancy found before declaring the system healthy.

## Script Maintenance Fixes
   - Removed duplicate `import time` statements
   - Fixed unterminated triple-quoted strings in docstrings
   - Added missing `import os` in systemd-managed scripts
   - Made systemd `ExecStart` paths explicit and absolute
   - Python `True`/`False` vs JSON `true`/`false` — Python boolean must be capitalized in models.py

## Diagnosis: "Local Model Not Connecting"

When the user reports local models aren't connecting, follow this structured triage:

### 1. Check the Three Layers (YAML, watchdog, Python switcher)
```
YAML       = model section in config.yaml
watchdog   = cron health checker (hermes cron list)
py switcher = auto-model-switch.py + hermes-config.path watcher
```

### 2. Config Check
```bash
grep -A 5 "^model:" ~/.hermes/config.yaml
# Verify: model.default, model.provider, model.base_url
```

### 3. Path Watcher Health
```bash
sudo systemctl is-active hermes-config.path
sudo journalctl -u hermes-config.path --no-pager
```
If `inactive (dead)`, the path watcher silently died. This is the #1 cause of models not switching properly — config changes are written correctly, but `auto-model-switch.py` never fires to stop old / start new services.

### 4. Systemd Unit Load Check
Before assuming a service failure, verify the unit file is actually known to systemd. New unit files added to `/etc/systemd/system/` require `sudo systemctl daemon-reload` before systemd recognizes them:
```bash
# Check if the unit is loaded at all
sudo systemctl list-units --all 'llama-server*'
# Missing from output → unit not loaded → needs daemon-reload

# If it IS loaded, check its state
sudo systemctl is-active llama-server-<model>
```

**Common scenario:** Unit file exists on disk (`/etc/systemd/system/llama-server-mtp.service`) but doesn't appear in `systemctl list-units`. This happens when the file was copied in place but `daemon-reload` was never run. The auto-switch script's `start_service()` silently fails (systemctl returns non-zero), the new model never starts, and Hermes hits "Connection error after 3 retries."

Fix:
```bash
sudo systemctl daemon-reload
sudo systemctl start llama-server-<model>
sudo systemctl status llama-server-<model>
```

**After daemon-reload, verify ALL llama-server units are visible:**
```bash
sudo systemctl list-units --all 'llama-server*'
# Every expected unit should appear as "loaded" (state may be inactive/dead)
```

### 5. Running Services Detection
```bash
# Check systemd unit states (system-level, not --user)
sudo systemctl list-units | grep -i llama-server
# Check for orphan processes (still alive even if systemd considers them dead)
ps aux | grep llama-server | grep -v grep
```

### 6. Auto-Switch Log
```bash
cat ~/.hermes/logs/model-switch.log | tail -20
```

### 7. State File
```bash
cat ~/.hermes/.last-model-state
# Shows model.default on first line, model.provider on second
```

### 8. Corrupt GGUF Detection
A corrupt GGUF file causes systemd to fail in a restart loop. Detect it:
```bash
# Check for failing services
sudo systemctl list-units | grep -i llama-server | grep -i "auto-restart\|failed"

# Read why it failed
journalctl -u <failing-service> --no-pager -n 15 | grep -i "corrupted\|error\|failed to load"

# Common corruption signal
```
tensor 'blk.27.ffn_down_exps.weight' data is not within the file bounds, model is corrupted or incomplete
```
Fix: redownload the GGUF file. The existing file is truncated/incomplete even if it has the right byte count.

### 9. Switch Log — Final Arbiter
```bash
cat ~/.hermes/logs/model-switch.log | tail -20
```
Shows every switch attempt, including which service was stopped/started, whether the API responded, and rollback attempts. A switch that "succeeds" (no crash) but shows "API ready: no in 60.0s" means the target service failed silently (corrupt model, wrong port, OOM).

### Expected Workflow (/model pick in Telegram/Hermes)
```
User in Telegram -> /model picks model
  |
  v
Gateway writes config.yaml via save_config()
  |
  v
hermes-config.path detects file change
  |
  v
hermes-config.service triggers auto-model-switch.py
  |
  v
auto-model-switch.py stops old llama-server (if local to local)
  |
  v
Waits 3s for VRAM drain (local to local only)
  |
  v
Starts new llama-server (if local)
  |
  v
Polls /health until model responds (up to 60s)
  |
  v
Sends "Model Loaded" notification via watchdog cron
```

If any step in this chain is broken, the switch silently fails — old processes stay alive, new ones never start, and the agent connects to the wrong port or a dead service.

### Diagnostic Signal: Auto-Switch Succeeded (Log Shows "API ready: yes") But Runtime Still Gets "Connection error"

This is a different failure mode from "auto-switch never ran." The auto-switch log shows MTP starting and responding on its port, but Hermes runtime retries and fails. This means one of:

1. **Systemd unit not loaded** — If the unit file was added but `daemon-reload` was never run, `start_service()` returns False, the script hits rollback (with no previous service = silent fail), and config retains the target model. Hermes runtime then tries to connect to the target port with nothing listening. Check: `sudo systemctl list-units --all 'llama-server-<target>'` — if missing, run `sudo systemctl daemon-reload` and retry.

2. **Gateway session override wrong base_url** — The `/model` picker writes `model.base_url` correctly to config.yaml, but the gateway's in-memory `_session_model_overrides` may cache `custom_providers[0].base_url` (e.g., port 8081 instead of 9022). This happens when the custom_providers grouping URL doesn't match the per-model port. Check: look at the gateway logs for what base_url was used in the API call. If it shows the wrong port, either `/reset` the session or add the model to `custom_providers` in config.yaml and verify the runtime resolver prefers `model.base_url` (see `_resolve_named_custom_runtime` in `runtime_provider.py`).

3. **Model in config.yaml but not in custom_providers** — The auto-switch script finds the target in `models.py` and starts the service, but if the model isn't listed in `custom_providers.models` in config.yaml, the `/model` picker can't properly resolve it. The gateway may not send the right session override. Fix: keep `custom_providers` in sync with `models.py`.

**Quick triage for this signal:**
```bash
# 1. Is the unit loaded?
sudo systemctl list-units --all 'llama-server-mtp' 2>/dev/null | grep -v '0 loaded'

# 2. Is the service actually running?
sudo systemctl is-active llama-server-mtp

# 3. Does the port actually respond?
curl -s -m 5 http://127.0.0.1:9022/v1/models | head -c 80

# 4. Does chat completion work?
curl -s -m 30 http://127.0.0.1:9022/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3.6-35B-A3B-UD-Q4_K_S.gguf","messages":[{"role":"user","content":"hi"}],"max_tokens":5,"temperature":0}' | python3 -c "import sys,json; print(json.load(sys.stdin).get('choices',[{}])[0].get('message',{}).get('content','TIMEOUT/FAIL'))"
```

If steps 1-3 pass but step 4 times out, you have a **hung llama-server** (see the `llama-cpp` skill's `references/hung-server-diagnosis.md`). Kill and restart.

**⚠️ Cloud switch gap:** When switching from a local model to a cloud provider, the
chain above stops at "Cloud model — no service to swap." The previously-running
llama-server is NOT stopped. Check for orphans afterward:

```bash
sudo systemctl list-units | grep llama-server.*running
```

## Common Pitfalls
- Duplicate time imports causing NameErrors
- Missing VRAM drain delay on local-to-local switches
- **Path watcher silently dies** — `hermes-config.path` can deactivate without notice. Always check `systemctl is-active hermes-config.path` first when local models wont connect.
- **Orphaned GPU processes** — When path watcher is dead, local-to-cloud switches leave old llama-server processes running, consuming VRAM. Detect via `ps aux | grep llama-server` (contrast with `sudo systemctl list-units | grep llama-server`).
- Premature service termination during cloud model transitions
- Unterminated triple-quoted strings in documentation
- Stopping CPU-only background services during GPU model switches (add `preserve: True`)
- ❌ Using `"service": "llama-server"` for multiple models — each needs its own unit file
- ❌ Leaving phantom model entries in models.py (defined but no .gguf on disk — causes silent failures)
- ❌ Port mismatch between models.py and systemd service files — models.py ports are just numbers that must match the --port flag in each service's ExecStart. Use the port-audit command in Verification Steps to cross-check.
- ❌ **Duplicate systemd service files on the same port** — Detect via the port-audit command and fix immediately before any switch attempt.
- ❌ **custom_providers drift from models.py** — models added to models.py but not to `custom_providers` in config.yaml won't appear in the /model picker. Conversely, models listed in custom_providers but missing from models.py will be selectable but no auto-switch runs on selection.
- ❌ **Corrupt GGUF causing silent restart loop** — systemd shows "auto-restart" for the service but the switch log shows "API ready: no in 60.0s". The model never loads. Detect via `journalctl -u <service> | grep -i "corrupted\|error loading model"`.
- ❌ **GGUF filenames in /model picker** — custom_providers uses GGUF filenames by default. The /model picker shows these raw filenames. If the user complains about ugly model names, implement **Option D** in `references/custom-providers-naming.md`: update auto-model-switch.py to match on both `name` and `model_name`, then switch custom_providers keys to friendly names.
- ❌ **Rapid-fire config writes bypass the watcher** — Writing `config.yaml` multiple times within the same second can give the same mtime, causing the path watcher to miss intermediate writes. The watcher's flock lock also prevents concurrent invocations, so a second write while the first swap is still in progress will be dropped entirely. Fix: always switch via `/model` in chat (atomic write), or add a minimum 3-second gap between manual config writes. When testing multiple switches in sequence, wait for the previous switch to complete before writing the next config.
- ❌ **Orphan services after cloud switch** — auto-model-switch.py only stops local services during local↔local transitions. Switching to a cloud provider logs "no service to swap" and returns, leaving the old llama-server running and consuming VRAM. Always check for orphans after a cloud switch: `sudo systemctl list-units | grep llama-server.*running`.
- ❌ **New unit files need `daemon-reload`** — Copying a `.service` file to `/etc/systemd/system/` does NOT automatically register it with systemd. Without `sudo systemctl daemon-reload`, `systemctl start llama-server-<name>` silently fails (exit code ≠ 0, no error to stderr). This causes the auto-switch script's `start_service()` to return False, rollback logic triggers, and the model never comes up. The auto-switch log shows `Switch complete: ... (API ready: no in 60.0s)` or no switch event at all. Fix: always run `sudo systemctl daemon-reload` after adding/removing/modifying unit files, then verify with `sudo systemctl list-units --all 'llama-server*'`.

## Verification Steps
1. `systemctl list-units | grep -i llama` - active service state
2. `model-switcher 35b` test with multiple local/cloud targets
3. `journalctl -u llama-server` - inspect service transitions
4. `nvidia-smi` - validate VRAM changes
5. **Port audit** — verify models.py ports match actual service files AND detect duplicate port assignments:
   ```bash
   for f in /etc/systemd/system/llama-server*.service; do
     name=$(basename "$f" .service)
     port=$(grep -oP -- '--port \K\d+' "$f" 2>/dev/null || echo "?")
     model=$(grep -oP -- '-m /\S+/\K[^/]+\.gguf' "$f" 2>/dev/null || echo "?")
     echo "$name: $model (port $port)"
   done | sort -t'=' -k2
   # Check for duplicate ports:
   echo "--- Duplicate ports? ---"
   for f in /etc/systemd/system/llama-server*.service; do
     port=$(grep -oP -- '--port \K\d+' "$f" 2>/dev/null)
     echo "$port $(basename $f)"
   done | sort | uniq -f1 -d
   ```
6. **Four-way alignment** — run the full audit in Core Technique #6 to verify disk ↔ models.py ↔ systemd ↔ custom_providers all agree, then fix any row that differs.
7. **Corrupt model scan** — check all running services for auto-restart loops:
   ```bash
   sudo systemctl list-units | grep llama-server.*auto-restart
   ```
8. **VRAM check** — confirm available memory matches expected model size:
   ```bash
   nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader
   ```

## Related References
- [Model switch error analysis](references/model-switch-errors.md)
- [Systemd lifecycle flow](references/systemd-management.md)
- [Script patch history](references/switcher-script-changes.md)
- [Custom providers naming in /model picker](references/custom-providers-naming.md)
- [Rapid-fire config write collision](references/rapid-fire-config-write-collision.md) — real-world mechanism behind swallowed writes during fast switching

## Reusable Scripts
- `scripts/verify-switches.py` — automated end-to-end test that runs a sequence of local↔local and local↔cloud switches, verifying path watcher, service, health, config persistence, and VRAM at each step. Use after any fleet change to confirm switching still works.

## Skill Evolution
Enhanced with fixes for VRAM-aware switching, systemd-specific handling, and script maintenance patterns from recent debugging sessions. Integrates lessons from solving import issues, string syntax errors, and service racing conditions. Focuses on making model switching atomic, reliable, and friendly to both GPU and configuration state.