# Multi-Model Hot-Swap

> ⚠️ **llama-swap is now the recommended approach.**  
> This systemd-based approach works but requires manual scripts and per-frontend config.
> [llama-swap](references/llama-swap.md) replaces all of this with a single binary, one config file, and zero manual swapping.
> Consider migrating unless you have a specific reason to keep per-port systemd services.
 with systemd + Hermes Agent

Complete guide for running multiple llama.cpp models on separate ports and hot-swapping between them without typing passwords.

## Architecture

```
┌────────────────────┐     ┌───────────────────┐
│ llama-server-35B   │     │  llama-server-9B   │
│ Port 8081          │     │  Port 8082          │
│ systemd service    │     │  systemd service    │
└────────┬───────────┘     └────────┬────────────┘
         │                         │
         └─────────┬───────────────┘
                   │
         Hermes Agent uses one at a time
         via custom_providers in config.yaml
```

## Prerequisite: Design Your Model's Flags

Before creating a systemd service, figure out the right flags for the model type and your hardware. See the **[Flag Design Guide](flag-design-guide.md)** for:

- VRAM budget math (GTX 1080 Ti = 11 GB)
- MoE vs Dense architecture flags
- Context sizing calculations
- Threading strategy for Xeon CPUs
- HuggingFace API workflow to fetch model config specs

## Setup Steps

### 1. Research Model Architecture (skip if already known)

Before writing configs, fetch each model's architecture from HuggingFace to determine MoE vs dense, layer count, head counts, and native context:

```bash
# Get architecture type: "Qwen3MoeForCausalLM" → MoE, "Qwen2ForCausalLM" → Dense
curl -sL "https://huggingface.co/<org>/<model>/raw/main/config.json" | \
  python3 -c "import json,sys; c=json.load(sys.stdin); print('layers:', c.get('num_hidden_layers'), 'heads:', c.get('num_attention_heads'), 'KV heads:', c.get('num_key_value_heads'), 'embed:', c.get('hidden_size'), 'context:', c.get('max_position_embeddings'), 'experts:', c.get('num_experts',0))"
```

Key decisions:
- **MoE models** (`Qwen3MoeForCausalLM`, `num_experts` > 0): Need `--n-cpu-moe N` where N = num_hidden_layers
- **Dense models**: Just `-ngl 99` to offload everything

### 2. Systemd Service Per Model

Each model gets its own service file in `/etc/systemd/system/` on a unique port. Copy flags from an existing working service and change:
- `-m /path/to/model.gguf` — model file path
- `--port NNNN` — unique port (e.g., 8081, 8082, 8083, etc.)
- `-c N / --ctx-size N` — context window for THIS model
- `--n-cpu-moe N` — only for MoE models, omit for dense
- `Description=` — update display name
- `WorkingDirectory=` — directory containing the model

**Flag template (dense model):**
```ini
ExecStart=/usr/local/bin/llama-server \\
  -m /models/path/to/model.gguf \\
  --port NNNN \\
  --host 0.0.0.0 \\
  --cache-type-k turbo4 \\
  --cache-type-v turbo4 \\
  -ngl 99 \\
  --no-mmap \\
  --mlock \\
  --jinja \\
  --prio 2 \\
  --poll 80 \\
  --cont-batching \\
  --timeout 300 \\
  -c N \\
  --ctx-size N \\
  --threads 16 \\
  --threads-batch 32 \\
  --parallel 1 \\
  --batch-size 4096 \\
  --ubatch-size 1024 \\
  --temp 0.2 \\
  --top-p 0.95 \\
  --min-p 0.05 \\
  --top-k 20 \\
  --metrics
```

**Flag template (MoE model) — add `--n-cpu-moe N`:**
```ini
  --n-cpu-moe 48 \
```

**Important service fields for all models:**
```ini
Restart=on-failure
RestartSec=5
LimitMEMLOCK=infinity
LimitNOFILE=1048576
```

**Systemd install:**
```bash
sudo cp ~/.hermes/llama-server-*.service /etc/systemd/system/
sudo systemctl daemon-reload
```

**Pitfall — MoE flags are model-specific:** `--n-cpu-moe` only applies to Mixture-of-Experts models. Do NOT copy it to non-MoE models or they will refuse to load with an error about unknown or incompatible flags.

### 3. Register in Hermes config.yaml

To keep the `/model` picker clean, consolidate all local models under a **single** `custom_providers` entry called `local` instead of making one entry per model:

```yaml
custom_providers:
- name: local
  base_url: http://127.0.0.1:8081/v1  # base_url gets rewritten by swap script
  models:
    Qwen3.6-35B-A3B-UD-Q4_K_S.gguf:
      context_length: 128000
    qwen3.5-9b-uncensored-Q4_K_M.gguf:
      context_length: 128000
    qwen3-8b-Q4_K_M.gguf:
      context_length: 65536
    Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf:
      context_length: 131072
    qwen3-14b-Q4_K_M.gguf:
      context_length: 65536
    gemma-3-12b-Q4_K_M.gguf:
      context_length: 131072
    gemma-4-26B-IQ4_XS.gguf:
      context_length: 131072
    gemma-3-12b-it-qat-Q4_0.gguf:
      context_length: 131072
    DeepSeek-R1-Distill-Llama-8B-Q5_K_M.gguf:
      context_length: 32768
    Ministral-3-8B-Reasoning-2512-Q5_K_M.gguf:
      context_length: 128000
    HuggingFaceTB_SmolLM3-3B-Q4_K_M.gguf:
      context_length: 16384
```

**Benefits of consolidation:**
- In `/model`, instead of 7 separate provider entries cluttering the list, you see one `local` provider with all models nested under it
- The swap script handles port changes by updating `model.base_url` — the provider name stays `custom:local`
- Adding new models just means a new entry in the `models:` dict

The `context_length` under each model should match the `-c` flag in the service file. The swap script updates `model.default`, `model.provider`, and `model.base_url` when switching.

## Auto-Switch Path Watcher

A systemd path watcher (`hermes-config.path`) triggers `auto-model-switch.py` whenever `~/.hermes/config.yaml` changes. This handles service-to-service swaps automatically:

- **Local → Local:** Stops current systemd service, starts target, fixes `model.base_url` to correct port, sends Telegram notification
- **Local → Cloud:** Does NOT stop the local service. The model stays warm in VRAM so switching back is instant. Only the cloud provider/model is set.
- **Cloud → Local:** Starts the target service since nothing is running locally
- **Rollback on failure:** If the target service fails to start, the script rolls back to the previous model and reverts config

The script uses a flock lock file (`~/.hermes/.model-switch.lock`) to prevent concurrent invocations from racing.

## Sudoers NOPASSWD (Empowers the Swap Script)

The swap script uses `sudo systemctl start/stop`. Create `/etc/sudoers.d/rurouni` to skip password prompts. Two approaches:

**Simple (recommended): catch-all NOPASSWD**
```bash
echo 'rurouni ALL=(ALL:ALL) NOPASSWD: ALL' > /tmp/rurouni-sudoers
sudo cp /tmp/rurouni-sudoers /etc/sudoers.d/rurouni
sudo chmod 440 /etc/sudoers.d/rurouni
rm /tmp/rurouni-sudoers
```
This lets `systemctl start/enable/stop all` work without TTY prompts and doesn't need updates when adding new services. Verified on this machine 2026-05-12 — without this, `sudo systemctl start` fails with "a terminal is required" even when the user approves the command.

**Alternative: per-command (more restrictive)**
Create `/etc/sudoers.d/hermes-models` listing each `systemctl start` and `systemctl stop` command individually.

**Pitfall:** Both `sudo systemctl start` and `sudo systemctl enable` need NOPASSWD. `sudo systemctl daemon-reload` and `sudo cp` usually work without it.

### 5. Model-Switcher Script

See [`templates/model-switcher`](../templates/model-switcher) — a Python TUI menu script that:
- Lists all models with active/current status, port, context, and use-case description
- Supports interactive menu (pick by number or shortcut name)
- Supports direct switch: `models 9b`, `models coder30`
- Stops the running model, starts the selected model, updates Hermes config, waits for the model to respond
- Each model entry in MODELS includes `desc` (one-liner), `detail` (long description), `short` (alias)

**Important when consolidated providers are used:** If you switched from separate `custom:X` providers to a single `custom:local`, the script's `get_current_config()` function must parse `model.default` (the filename) instead of `model.provider` (which is always `custom:local`). The template script uses model-name matching, which works with either approach.

Edit the `MODELS` list in the script to add/remove entries for your setup.

**Shell alias:** Add to `~/.bashrc`:
```bash
alias models="python3 /home/rurouni/.hermes/scripts/model-switcher"
```

Usage:
```bash
models                                # interactive TUI menu
models 9b                             # quick switch by shortcut
models 35b                            # direct switch by name
models coder30                        # switch to Coder-30B
```

## Hot-Swap Commands (Manual, without script)

```bash
# Port map (current as of May 2026):
#   8081 = llama-server            (Qwen3.6-35B-A3B)
#   8082 = llama-server-9b         (Qwen3.5-9B-Uncensored)
```bash
# Port map (current as of May 2026):
#   8081 = llama-server            (Qwen3.6-35B-A3B)
#   8082 = llama-server-9b         (Qwen3.5-9B-Uncensored)
#   8083 = llama-server-ds-r1-8b   (DeepSeek-R1-Distill-Llama-8B)
#   8084 = llama-server-ministral-8b (Ministral-3-8B-Reasoning-2512)
#   8086 = llama-server-qwen8b     (Qwen3-8B)
#   8087 = llama-server-smol       (SmolLM3-3B — CPU only, always running for Hindsight)
#   8088 = llama-server-qwen14b    (Qwen3-14B)
#   8089 = llama-server-gemma3     (Gemma 3 12B)
#   8090 = llama-server-gemma4     (Gemma 4 26B MoE)
#   8091 = llama-server-gemma3-qat (Gemma 3 12B QAT)
#   8092 = llama-server-qwen4b     (Qwen3-4B-Instruct-2507)

# NOTE: Manual commands shown for reference. Prefer the model-switcher script
# for routine use — it does a single atomic YAML write instead of 3 separate
# `hermes config set` calls, avoiding path-watcher races.

# Switch to 35B
sudo systemctl stop llama-server-9b.service
sudo systemctl start llama-server.service
hermes config set model.default Qwen3.6-35B-A3B-UD-Q4_K_S.gguf
hermes config set model.provider custom:local
hermes config set model.base_url http://127.0.0.1:8081/v1

# Switch to 9B
sudo systemctl stop llama-server.service
sudo systemctl start llama-server-9b.service
hermes config set model.default qwen3.5-9b-uncensored-Q4_K_M.gguf
hermes config set model.provider custom:local
hermes config set model.base_url http://127.0.0.1:8082/v1

# Switch to 14B
sudo systemctl stop llama-server-9b.service
sudo systemctl start llama-server-qwen14b.service
hermes config set model.default qwen3-14b-Q4_K_M.gguf
hermes config set model.provider custom:local
hermes config set model.base_url http://127.0.0.1:8088/v1

# Switch to Gemma 3 QAT
sudo systemctl stop llama-server-qwen14b.service
sudo systemctl start llama-server-gemma3-qat.service
hermes config set model.default gemma-3-12b-it-qat-Q4_0.gguf
hermes config set model.provider custom:local
hermes config set model.base_url http://127.0.0.1:8091/v1
```

## Verification

After switching, verify the new model is serving:

```bash
curl http://127.0.0.1:8081/v1/models   # replace port as needed
nvidia-smi                              # check VRAM usage
systemctl status llama-server.service   # replace service as needed
```

## Adding a New Model (Checklist)

1. Download the `.gguf` file to `~/models/llm/<category>/`
2. Research architecture via HuggingFace config.json (MoE or dense?)
3. Calculate context window using VRAM math (see flag-design-guide.md)
4. Create systemd service file with correct flags
5. Copy to `/etc/systemd/system/` and `systemctl daemon-reload`
6. Add model entry to `custom_providers.local.models:` in Hermes config.yaml
7. Add to sudoers file for NOPASSWD
9. Add entry to MODELS list in `templates/model-switcher`
10. Copy script to `~/.hermes/scripts/` and test

## Key Pitfalls

- **VRAM drain delay:** When stopping a GPU model and immediately starting another, the NVIDIA driver takes 5-15s to fully reclaim VRAM after the process exits. The new service's allocation may fail or hang if it tries to claim VRAM that's still in a `Released` state. For large models (26B MoE on 11GB), this is especially likely. **Fix:** Add `sleep 10` between stop and start, or poll `nvidia-smi` for free VRAM to return to expected levels before proceeding.
- **MoE + VRAM contention:** Gemma-4-26B-A4B (IQ4_XS, ~11GB model weight + overhead) may fail to load within 60s after stopping a large dense model (Qwen3-14B) because VRAM hasn't fully cleared. Symptoms: `systemctl start` reports success, but API never responds (health check times out). Check `journalctl -u llama-server-gemma4.service` for allocation errors.
- **Cloud switch keeps local warm:** The auto-switch script explicitly does NOT stop local services when going to cloud models. If local services are dead after a cloud switch, something else killed them (OOM, crash, manual stop, or the previous local→local swap left a dead service). Don't blame the cloud switch.
- **Each model needs a unique port** — they cannot share.
- **Auto-switch race on quick config changes:** The path watcher (`hermes-config.path`) triggers on every YAML write. A single atomic write triggers once. Three `hermes config set` calls trigger it three times — each sees partial state and may race. Always prefer a single atomic YAML update that writes all three model fields at once.