---
name: new-model-onboarding
description: "Step-by-step checklist for adding a new GGUF model to Ray's current 1080 Ti local Hermes stack without breaking switching or the /model menu."
version: 2.0.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [llama-cpp, gguf, local-models, hermes, systemd]
    related_skills: [local-model-switch-optimization, hermes-agent, llama-cpp]
---

# New Model Onboarding

## Overview

Use this when adding a **new local GGUF model** to Ray's Hermes stack on the Dell Precision 5810 / GTX 1080 Ti 11GB machine. The goal is to add a model cleanly without breaking:
- systemd service startup
- Hermes `/model` switching
- the `custom:local` provider menu
- long-context stability on 11GB VRAM

This skill reflects the **current trimmed fleet** and should be updated whenever the active lineup changes.

## Current Ground Truth

Current local fleet:
- `Qwen3.6-35B-A3B-UD` → port `8081` → service `llama-server.service`
- `Qwen3.5-9B-UD` → port `8082` → service `llama-server-9b.service`
- `Gemma-4-26B` → port `8090` → service `llama-server-gemma4.service`

Also still present but not part of the chat-model fleet:
- `/models/llm/memory/HuggingFaceTB_SmolLM3-3B-Q4_K_M.gguf`
- `/models/embeddings/Qwen3-Embedding-0.6B/Qwen3-Embedding-0.6B-Q8_0.gguf`

The `/model` picker's OpenRouter block is curated by:
- `~/.hermes/cache/openrouter_override.json`

## When to Use

Use this skill when:
- downloading a new GGUF you want selectable from Hermes `/model`
- replacing one of the 3 kept local models
- adding a new systemd llama-server unit
- checking whether a candidate fits 60K-100K+ context on the 1080 Ti

Do **not** use this skill for:
- cloud-only models with no local GGUF
- one-off benchmark experiments that won't join the rotation
- debugging an already-installed model switch failure — use `local-model-switch-optimization`

## Core Rules

1. **One active chat model at a time.** Design for clean switching, not concurrent GPU fleet usage.
2. **Every local model needs its own systemd unit** with the real GGUF path and real port.
3. **All four layers must agree:** disk ↔ systemd unit ↔ `~/.hermes/scripts/models.py` ↔ `custom_providers.local.models` in `~/.hermes/config.yaml`.
4. **Local switching relies on `custom:local`.** Keep `model.provider` stable and move `model.base_url` to the selected port.
5. **After any local model change, verify the path watcher still works:** `systemctl is-active hermes-config.path`.

## 1. Preflight

Verify prerequisites before downloading anything:

```bash
sudo -n true
systemctl is-active hermes-config.path
python3 ~/.hermes/scripts/models.py
```

You should confirm:
- NOPASSWD sudo works
- the path watcher is active
- the current fleet inventory prints cleanly

## 2. Download and Place the GGUF

Store the model under a sensible category:

```bash
wget -O /models/llm/<category>/<filename>.gguf "<download-url>"
```

Suggested categories:
- `/models/llm/daily/` — flagship / best general model
- `/models/llm/uncensored/` — uncensored general chat
- `/models/llm/gemma4/` — Gemma 4 family
- `/models/llm/<new-family>/` — any new family you want kept distinct

If the model needs vision support, also place its matching `mmproj` file beside it.

## 3. Fit Check for 1080 Ti

Before wiring it into Hermes, benchmark the model by itself.

### Practical guidance from current fleet
- `Qwen3.6-35B-A3B-UD` works at `128K` with `4096/1024`
- `Qwen3.5-9B-UD` works at `128K` with `1024/256`
- `Gemma-4-26B` works at `131072` with `2048/512`

### Long-context rules learned on this machine
- Minimum acceptable target for local work: **60K**
- Prefer **100K+** where the model can actually hold it
- `ubatch-size` is the main anti-OOM knob on this hardware
- `turbo4` KV cache is the default first choice
- `--no-kv-offload` is often required for dense models at high context
- Flash attention support is model/build dependent; don't assume from docs alone — test it

## 4. Pick a Free Port and Service Name

Current occupied ports:
- `8081`
- `8082`
- `8090`

Pick a free port outside those. Then create a dedicated unit:
- `/etc/systemd/system/llama-server-<name>.service`

Do **not** reuse another model's unit.

## 5. Create the Systemd Unit

Write the unit to `/tmp` first, then install it under `/etc/systemd/system/`.

Dense-model baseline template:

```ini
[Unit]
Description=<Model Name> llama-server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=root
WorkingDirectory=/models/llm/<category>
ExecStart=/usr/local/bin/llama-server \
  -m /models/llm/<category>/<filename>.gguf \
  --port <PORT> --host 0.0.0.0 \
  --cache-type-k turbo4 --cache-type-v turbo4 \
  -ngl 99 --no-mmap --mlock --jinja --no-kv-offload \
  --chat-template-kwargs '{"enable_thinking":false}' \
  --prio 2 --poll 80 --cont-batching --timeout 300 \
  -c <CTX> --ctx-size <CTX> \
  --threads 16 --threads-batch 32 \
  --parallel 1 --batch-size <BATCH> --ubatch-size <UBATCH> \
  --temp 0.2 --top-p 0.95 --min-p 0.05 --top-k 20 \
  --metrics
Restart=on-failure
RestartSec=5
LimitMEMLOCK=infinity
LimitNOFILE=1048576

[Install]
WantedBy=multi-user.target
```

MoE models may need `--n-cpu-moe`. Vision models may need `--mmproj ...`.

## 6. Verify the Unit in Isolation

Before touching Hermes config, test the service directly:

```bash
sudo systemctl daemon-reload
sudo systemctl start llama-server-<name>.service
curl -s http://127.0.0.1:<PORT>/health
journalctl -u llama-server-<name>.service --no-pager -n 40
nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader
```

Do not proceed until:
- `/health` returns ok
- journal shows the model loaded successfully
- VRAM use is plausible

## 7. Register the Model in Hermes

### A. Add to `~/.hermes/scripts/models.py`

Use the real friendly name, GGUF filename, port, context, service, and alias:

```python
{
    "name": "Human-Readable-Model-Name",
    "desc": "Short summary",
    "detail": "Longer note with context / strengths / constraints.",
    "provider": "local",
    "model_name": "<filename>.gguf",
    "port": <PORT>,
    "context": <CTX>,
    "service": "llama-server-<name>",
    "short": "<alias>",
}
```

### B. Add to `custom_providers.local.models` in `~/.hermes/config.yaml`

```yaml
custom_providers:
- name: local
  base_url: http://127.0.0.1:8081/v1
  models:
    ...existing models...
    Human-Readable-Model-Name:
      context_length: <CTX>
```

Keep the model key aligned with the friendly `name` you want visible in `/model`.

## 8. Verify the Full Switch Path

After registration, test the real Hermes switching chain:

```bash
systemctl is-active hermes-config.path
python3 ~/.hermes/scripts/models.py
```

Then switch to the model via Hermes `/model` or the supported switcher path and verify:
- config updates all 3 fields: `model.default`, `model.provider`, `model.base_url`
- the correct service starts
- the service answers on its assigned port
- the gateway talks to that port, not the grouping URL by mistake

## 9. If the New Model Replaces an Old One

Remove the replaced model from all four places:
- delete the GGUF / mmproj from disk
- remove the systemd unit
- remove the `models.py` entry
- remove the `custom_providers.local.models` entry

This stack should stay intentionally small; keep only models with a real niche.

## Common Pitfalls

1. **Only editing one layer.** If disk, systemd, `models.py`, and `custom_providers` drift, the picker and switcher disagree.
2. **Using a generic shared unit.** The wrong GGUF starts even though the switch appears to work.
3. **Keeping stale models in `custom_providers`.** The `/model` menu shows dead options.
4. **Using too-large batch/ubatch at high context.** On this machine, reducing `ubatch-size` is often what saves a borderline model.
5. **Verifying OpenRouter models with `get_catalog()` only.** If an override file is configured, the picker follows `get_curated_openrouter_models()` instead.
6. **Forgetting the path watcher.** If `hermes-config.path` is inactive, config changes persist but no swap happens.

## Verification Checklist

- [ ] GGUF downloaded to the intended path
- [ ] Optional mmproj present if required
- [ ] Dedicated systemd unit created
- [ ] Service starts and `/health` returns ok
- [ ] Port is unique
- [ ] `models.py` entry added and syntax-valid
- [ ] `custom_providers.local.models` entry added
- [ ] `/model` switch updates provider/default/base_url atomically
- [ ] The gateway hits the correct per-model port
- [ ] Any replaced model was removed from disk, systemd, `models.py`, and `config.yaml`