# New Model Onboarding Checklist

Step-by-step process for adding a new GGUF model to Ray's local stack (Dell Precision 5810, GTX 1080 Ti 11GB VRAM, 62GB RAM, 16C/32T Xeon, PCIe 3.0).

## Core Principle

**You only run ONE model at a time** — the model-switcher stops the current service before starting the next. Don't think in terms of "sharing VRAM between models." The full 11GB is available for whichever model is active.

## 0. Prerequisite: NOPASSWD sudoers

All model management uses `sudo systemctl start/stop`. Without `/etc/sudoers.d/rurouni`, these fail with "a terminal is required to read the password" even when the user approves the command.

```bash
# Check
sudo cat /etc/sudoers.d/rurouni 2>/dev/null || echo "MISSING"

# Fix
echo 'rurouni ALL=(ALL:ALL) NOPASSWD: ALL' > /tmp/rurouni-sudoers
sudo cp /tmp/rurouni-sudoers /etc/sudoers.d/rurouni
sudo chmod 440 /etc/sudoers.d/rurouni
rm /tmp/rurouni-sudoers
```

## 1. Download

```bash
# Download to /models/llm/<category>/
wget -O /models/llm/<category>/<filename>.gguf "<hf-download-url>"
```

**Category conventions:**
- `/models/llm/daily/` — general-purpose drivers (Qwen3.6-35B)
- `/models/llm/phi/` — Phi family (Phi-4 14B)
- `/models/llm/gemma3/` — Gemma family (Gemma 3 12B)
- `/models/llm/memory/` — small background models (SmolLM3-3B)
- `/models/llm/router/` — tiny routing/classification (Llama-3.2-3B)
- `/models/llm/uncensored/` — abliterated (Qwen3.5-9B)

## 2. Determine VRAM Fit

Only ONE model runs at a time (via model-switcher), so full 11GB is available.

### GGUF size estimates at Q4_K_M

| Params | Quant | Model Size | Total VRAM* | Fit |
|--------|-------|-----------|-------------|-----|
| 12B | Q4_K_M | ~6.8 GB | ~7.7 GB | ✅ Comfy (3.3 GB free) |
| 14B | Q4_K | ~8.5 GB | ~8.8 GB | ✅ Good (2.2 GB free) |
| 24B | Q3_K_M | ~11.5 GB | — | ❌ Too large for 11GB |
| 7-8B | Q5_K_M | ~5.5 GB | ~6-7 GB | ✅ Comfy |

*Total VRAM = model + CUDA overhead + compute buffers. With `--no-kv-offload`, KV cache is on CPU RAM, so the model's full weight fits in VRAM.

### `--fa` (flash attention) — GPU compute capability matters

- **GTX 1080 Ti (CC 6.1):** `--fa` flag gives `error: invalid argument: --fa`. Always omit.
- **Gemma 3 12B quirk:** Even though the `--fa` flag is rejected, llama-server **auto-detects and enables FA automatically** for Gemma 3 (logs: "Flash Attention was auto, set to enabled"). This works because Gemma 3's GQA architecture has FA kernels compiled for CC 6.1, while Phi-4's doesn't.

**Rule:** A model can support FA even when `--fa` flag fails — check server logs for "Flash Attention was auto, set to enabled."

### VRAM budgeting (14B dense example)

```
Free VRAM = 11,264 MiB - model_size - CUDA_overhead
For Phi-4 14B at -ngl 99:
  Model:        8,821 MiB
  Overhead:      ~300 MiB  
  Compute buf:  ~695 MiB (turbo4) or 1,440 MiB (f16)
  Free:        ~1,400 MiB (not enough for 64K KV)
  
Solution: --no-kv-offload → KV on CPU (62GB available), model on GPU
Result: ~25 t/s — no measurable slowdown
```

## 3. Benchmark Before Adding to Rotation

**Setup:** Stop all existing services, start temp server:

```bash
sudo pkill -f "llama-server"
/usr/local/bin/llama-server-turbo \
  -m /path/to/model.gguf --port 8090 --host 127.0.0.1 \
  -ngl 99 --no-mmap --jinja --no-kv-offload --cont-batching \
  --cache-type-k turbo4 --cache-type-v turbo4 \
  -c 65536 --ctx-size 65536 \
  -t 16 --threads-batch 32 --batch-size 2048 --ubatch-size 512 \
  --temp 0.2 --metrics 2>&1 &
```

**Wait for load** (~60s for 8.5GB over PCIe 3.0, ~30s for 6.8GB):

```bash
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8090/v1/models
# Expect: 200
```

### Speed Benchmark (using /metrics endpoint)

The `/metrics` endpoint gives accurate cumulative counters. This is more reliable than the timings field in `/chat/completions`:

```bash
# Write benchmark script
cat > /tmp/bench.py << 'EOF'
import json, urllib.request, time

def get_metrics():
    resp = urllib.request.urlopen("http://127.0.0.1:PORT/metrics", timeout=5)
    text = resp.read().decode()
    result = {}
    for line in text.split("\n"):
        parts = line.split()
        if len(parts) >= 2 and parts[0].startswith("llamacpp:"):
            try: result[parts[0]] = float(parts[-1])
            except: pass
    return result

data = json.dumps({"messages":[{"role":"user","content":"Write a paragraph about AI."}],"max_tokens":200,"temperature":0.2,"stream":False}).encode()

for i in range(2):  # warmup
    m1, req = get_metrics(), urllib.request.Request("http://127.0.0.1:PORT/v1/chat/completions", data=data, headers={"Content-Type":"application/json"})
    urllib.request.urlopen(req, timeout=60)

for i in range(3):  # real runs
    m1 = get_metrics()
    req = urllib.request.Request("http://127.0.0.1:PORT/v1/chat/completions", data=data, headers={"Content-Type":"application/json"})
    resp = urllib.request.urlopen(req, timeout=60)
    result = json.loads(resp.read())
    m2 = get_metrics()
    
    pt = int(m2.get("llamacpp:prompt_tokens_total",0) - m1.get("llamacpp:prompt_tokens_total",0))
    ps = m2.get("llamacpp:prompt_seconds_total",0) - m1.get("llamacpp:prompt_seconds_total",0)
    gt = int(m2.get("llamacpp:tokens_predicted_total",0) - m1.get("llamacpp:tokens_predicted_total",0))
    gs = m2.get("llamacpp:tokens_predicted_seconds_total",0) - m1.get("llamacpp:tokens_predicted_seconds_total",0)
    
    print(f"Run {i+1}: Prompt {pt}@{(pt/ps):.0f} t/s | Gen {gt}@{(gt/gs):.1f} t/s")
EOF

sed 's/PORT/8090/g' /tmp/bench.py | python3
```

**Baseline speed targets:** Qwen3.6-35B MoE = 10-12 t/s | Phi-4 14B = ~25 t/s | Gemma 3 12B = ~20 t/s | DeepSeek Flash = 50+ t/s | Anything below 8 t/s is too slow for chat.

**VRAM after model loads:**

```bash
nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader
```

### Config Variant Testing

Test at least these configurations and pick the best:

| Variant | Flag Changes | Test |
|---------|-------------|------|
| A: Default cache | Omit `--cache-type-k/v` | Compare gen speed, check VRAM |
| B: turbo4 | `--cache-type-k turbo4 --cache-type-v turbo4` | Usually best — saves VRAM, speeds gen |
| C: Big batch | `--batch-size 4096 --ubatch-size 1024` | Test if prompt speed improves |
| D: 128K ctx | `-c 131072 --ctx-size 131072` | For models with native 128K |

**Empirical finding:** On GTX 1080 Ti, turbo4 consistently beats default f16 (24% faster gen for Gemma 3, 25% for Phi-4). Increasing batch beyond 2048/512 doesn't improve speed but uses more VRAM — stick with 2048/512.

## 4. Find a Free Port

Current fleet (after May 2026 cleanup):
- 8081: Qwen3.6-35B
- 8082: Qwen3.5-9B  
- 8086: Llama-3.2-3B
- 8087: SmolLM3-3B
- 8088: Phi-4 14B
- 8089: Gemma 3 12B

Next free: **8090**

**Removed:**
- Qwen3-8B (port 8084) — outclassed by Qwen3.5-9B
- Qwen3-Coder-30B-A3B (port 8083) — too slow on CPU offload
- Qwen2.5-Coder-7B (port 8085) — outpaced by Phi-4 + Gemma 3

## 5. Create Systemd Service

Match the exact flag style of existing services. Write to `/tmp/` first, then `sudo cp`:

```ini
[Unit]
Description=<Name> llama-server-turbo
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=root
WorkingDirectory=/models/llm/<category>
ExecStart=/usr/local/bin/llama-server-turbo \
  -m /models/llm/<category>/<filename>.gguf \
  --port <PORT> --host 0.0.0.0 \
  --cache-type-k turbo4 --cache-type-v turbo4 \
  -ngl 99 --no-mmap --mlock --jinja \
  --no-kv-offload \
  --prio 2 --poll 80 --cont-batching --timeout 300 \
  -c <CTX> --ctx-size <CTX> \
  --threads 16 --threads-batch 32 \
  --parallel 1 --batch-size 2048 --ubatch-size 512 \
  --temp 0.2 --top-p 0.95 --min-p 0.05 --top-k 20 \
  [--rope-scale <FACTOR>] \
  --metrics
Restart=on-failure
RestartSec=5
LimitMEMLOCK=infinity
LimitNOFILE=1048576

[Install]
WantedBy=multi-user.target
```

**Flag rules:**
- `--no-kv-offload` **required** for 64K+ context on dense models. KV on CPU = ~100MB in system RAM. Without it: ~3.4GB+ VRAM for KV at 64K — won't fit.
- No `--fa` flag (CC 6.1 incompatible). Gemma 3 auto-detects FA regardless.
- `--rope-scale` only if model's native context < target context. Phi-4 native=16K, `--rope-scale 8.0` = 131K (enough for any 64K+ target).
- Batch size 2048/512 is optimal on this hardware (4096/1024 uses more VRAM with no speed gain).
- No `--n-cpu-moe` for dense models. Only for MoE (Qwen3.6-35B, Qwen3-Coder-30B).

## 6. Enable and Start

```bash
sudo cp /tmp/llama-server-<name>.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo ln -sf /etc/systemd/system/llama-server-<name>.service \
  /etc/systemd/system/multi-user.target.wants/
sudo systemctl start llama-server-<name>.service
```

**Load time:** ~60s for 8.5GB, ~30s for 6.8GB. Check readiness:
```bash
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:<PORT>/v1/models
```

## 7. Update models.py

Add to `~/.hermes/scripts/models.py`:

```python
{
    "name": "Human-Name",
    "desc": "emoji DESCRIPTION: summary for model-switcher display",
    "detail": "Longer description with VRAM, speed, context, and unique strengths.",
    "service": "llama-server-<name>.service",
    "port": <PORT>,
    "provider": "custom:local",
    "model_name": "<filename>.gguf",
    "context": <CTX>,
    "short": "<alias>",
}
```

## 8. Verify with model-switcher

**This is the definitive test** — the model-switcher is how Ray actually uses models:

```bash
python3 ~/.hermes/scripts/model-switcher <alias>
```

Check output for:
- "🔄 Switching to <Name>..." ✅
- "▶️ Starting <Name> ✅" — service started
- "⚙ Config model.provider / model.default / model.base_url" — config updated
- "⏳ Waiting... ✅" — health check passed
- Model runs `/reset` in Hermes to start a session with the new model

## 9. Fleet Cleanup (When to Remove Models)

A model is redundant when it no longer fills a unique slot:

| Signal | Real Example |
|--------|-------------|
| Same size class, strictly worse | Qwen3-8B → Qwen3.5-9B (newer, uncensored, 64K vs 41K) |
| Too big for GPU, slower on CPU | Coder-30B (17GB, needs CPU offload → ~8 t/s) |
| Outpaced by generalist | Coder-7B → Phi-4 (both fit on GPU, Phi-4 codes better) |
| No unique capability added | Any model that doesn't add vision, 1M ctx, multilingual, or fast reasoning |

**Cleanup procedure:**
```bash
# 1. Stop & remove service
sudo systemctl stop llama-server-<name>.service
sudo rm -f /etc/systemd/system/llama-server-<name>.service
sudo rm -f /etc/systemd/system/multi-user.target.wants/llama-server-<name>.service
sudo systemctl daemon-reload

# 2. Delete model file
rm -f /models/llm/<category>/<filename>.gguf

# 3. Remove from models.py
# Edit ~/.hermes/scripts/models.py — delete the entry

# 4. Update this skill's Current Fleet section
## Models Onboarded

| Model | Quant | Size | Port | VRAM | Speed | Context | Notes |
|-------|-------|------|------|------|-------|---------|-------|
| Phi-4 14B | Q4_K | 8.5GB | 8088 | 8.8GB | 25 t/s | 65K | Rope-scaled 16K→65K, dense, MS arch |
| Gemma 3 12B | Q4_K_M | 6.8GB | 8089 | 7.7GB | 20 t/s | 128K | Native 128K, Google arch, FA auto-enabled |

**Removed:** Mistral Small 3.1 24B (too large, ~9 t/s) | Phi-4-mini 5B (accidental download) | Qwen3-8B (outclassed by 9B) | Qwen3-Coder-30B (too slow on offload) | Qwen2.5-Coder-7B (outpaced by Phi-4 + Gemma 3)