# llama-swap — Config Reference

## Global Options

### sendLoadingState (default: true)

When `true`, llama-swap immediately streams loading progress via `reasoning_content` while the model loads from cold. Without this, the first request to a cold model hangs silently for 30-70s.

**Pitfall:** Chat UIs that display `reasoning_content` (Open WebUI, etc.) render these loading banners as "Thought for X seconds" with joke text like *"Allocating memory like a billionaire allocates tax avoidance strategies"*. This makes it look like the model is thinking when it's actually just loading.

**Recommendation:** Set to `false` for reasoning-aware UIs. The silent hang is preferable to fake "thinking" spam. Toggle to `true` only when debugging loading times or using a non-reasoning client that needs load feedback.

### healthCheckTimeout (default: 120s, min 15s)
Max seconds to wait for a model's `/health` endpoint. Models on a 2080 Ti can take 31-67s to load (22GB MoE). Default 120s is sufficient.

### performance monitoring
```yaml
performance:
  disabled: false
  every: 15s
```
Records GPU stats via `nvidia-smi`. Set to 15s to reduce polling noise.

## Canonical Model Config (MoE on 2080 Ti)
```yaml
--cache-type-k turbo3 --cache-type-v turbo4
--spec-draft-n-max 4
--spec-draft-type-k q4_0 --spec-draft-type-v q4_0
-ngl 99
--no-mmap --mlock
--jinja
--chat-template-kwargs '{"enable_thinking":false}'
--prio 2 --poll 80 --cont-batching
--timeout 300
-c 128000 --ctx-size 128000
--threads 16 --threads-batch 32
--n-cpu-moe 128
--parallel 1 --batch-size 2048 --ubatch-size 512
--temp 0.2 --top-p 0.95 --min-p 0.05 --top-k 20
--metrics
```

## Multi-Build / Custom Fork Pattern
Install fork to dedicated path: `/usr/local/lib/prism/` (for .so files), `/usr/local/bin/llama-server-prism` (binary).

Add systemd env override:
```bash
mkdir -p /etc/systemd/system/llama-swap.service.d
cat > /etc/systemd/system/llama-swap.service.d/env.conf << 'EOF'
[Service]
Environment=LD_LIBRARY_PATH=/usr/local/lib/prism
EOF
systemctl daemon-reload
```

Model entry:
```yaml
bonsai-8b:
  cmd: /usr/local/bin/llama-server-prism -m /models/downloads/Bonsai-8B-Q1_0.gguf --port ${PORT} --host 127.0.0.1 --jinja -ngl 99 --no-mmap --no-host -c 2048 -t 1 --temp 0.2
```

**Important:** Use single-line `cmd:` string (not YAML `>` folded block). The `env` prefix in YAML cmd does NOT work — Go's os/exec doesn't parse it. Always use systemd `Environment=` drop-in.

## Registered Models — 2080 Ti (11GB VRAM, SM75)

| Model | Context | t/s | VRAM | TTL | Type |
|-------|---------|-----|------|-----|------|
| nex-n2-mini | 250K | 19 | 10.8GB | 1800 | 35B MoE |
| qwen36-35b-mtp ⭐ | 250K | 48.6 | 5.0GB | 3600 | 35B MoE+MTP |
| qwopus-9b | 150K | 45.8 | 8.1GB | 1800 | 9B dense+MTP |
| qwen35-9b-mtp | 250K | 45.7 | 7.4GB | 1800 | 9B dense+MTP |
| gemma-12b | 256K | 53.7 | 8.4GB | 3600 | 12B dense |
| gemma-26b | 256K | 35.4 | 9.0GB | 1800 | 26B MoE |
| bonsai-8b | 65K | 150 | 1.0GB | 1800 | 8B Q1_0 |

## Current Model IDs in Hermes config.yaml

| llama-swap key | Hermes id | context_length |
|---|---|---|
| `qwen36-35b-mtp` | `qwen36-35b-mtp` | 250000 |
| `qwopus-9b` | `qwopus-9b` | 175000 |
| `qwen35-9b-mtp` | `qwen35-9b-mtp` | 250000 |
| `gemma-12b` | `gemma-12b` | 256000 |
| `gemma-26b` | `gemma-26b` | 256000 |
| `nex-n2-mini` | `nex-n2-mini` | 250000 |
| `bonsai-8b` | `bonsai-8b` | 65536 |

## Frontend Integration

**Hermes Agent custom_providers:**
```yaml
custom_providers:
  - name: local
    base_url: http://127.0.0.1:9292/v1
    models:
      Qwen3.6-35B-A3B-UD: { context_length: 128000 }
      Qwen3.6-35B-MTP:    { context_length: 128000 }
      ...
```

**LibreChat:** `baseURL: "http://host.docker.internal:9292/v1"`
**OpenCode (CachyOS):** `baseURL: "http://192.168.1.50:9292/v1"`

## Context Scaling (11GB)
- **MoE models:** Easily handle 128K+ at Q4+ (experts in system RAM via `--n-cpu-moe 128`)
- **Dense models (9B):** 128K fine with `--no-kv-offload`
- **Dense 14B:** VRAM bottleneck — ~7GB weights + ~5GB KV = ~12GB > 11GB. Stuck at 65K.

## Cross-config Sync Check
After any llama-swap config change:
```bash
# What llama-swap is configured to serve
grep -E '\-c |--ctx-size' ~/.config/llama-swap/config.yaml
# What's actually running
curl -s http://localhost:9292/running | python3 -c "import sys,json; d=json.load(sys.stdin); [print(r['cmd']) for r in d.get('running',[])]"
# What Hermes thinks
grep -A1 -E 'context_length:' ~/.hermes/config.yaml
```

## Model Aliases
- nex-n2-mini → nex-n2, Nex-N2-mini
- qwen36-35b-mtp → qwen36, qwen36-mtp, Qwen3.6-35B-MTP-unsloth
- qwopus-9b → qwopus, coder-9b, Qwopus3.5-9B-Coder
- qwen35-9b-mtp → uncensored, qwen35-ud, qwen35-9b
- gemma-12b → gemma-12b, gemma4-12b
- gemma-26b → gemma, gemma-4-26b
- bonsai-8b → bonsai, bonsai-8b, Bonsai-8B