---
name: llama-server-flags
description: Battle-tested llama-server flags for RTX 2080 Ti (sm_75, Turing, 11GB) — b9743 unified binary, flag templates, build recipes.
---

# llama-server Flag Templates

**Target:** RTX 2080 Ti 11GB, CUDA sm_75 (Turing), CUDA 12.6, g++-13

## Build Selection
| Build | Binary | Use for |
|-------|--------|---------|
| **b9743** (current) | `llama-server-sm75` | **All profiles** — unified. Replaces old sm75 (b9341 fork) + upstream (ac4cdde). |

**Build recipe:**
```bash
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=75 \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-13 \
  -DCMAKE_C_COMPILER=/usr/bin/gcc-13 -DCMAKE_CXX_COMPILER=/usr/bin/g++-13 \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --config Release --target llama-server
sudo cp build/bin/llama-server /usr/local/bin/llama-server-sm75
sudo cp build/bin/*.so* /usr/local/lib/ && sudo ldconfig
```

**Key differences from old (b9341/ac4cdde):** No TurboQuant (q8_0 floor), Eagle3 available, MMVQ_TURING, `mtp`→`draft-mtp`, 3s cold start, more VRAM-efficient.

**Applied patches (Jul 2026):** Two experimental MoE prefetch patches from thecodacus/llama.cpp (`fable5/prefetch-experts`) are cherry-picked onto b9743 — see `references/fable5-moe-prefetch-patches.md`. The patches are opt-in via `GGML_SCHED_PREFETCH_EXPERTS=1` and are **disabled by default**. On our RTX 2080 Ti (Turing, PCIe Gen3, `--no-mmap`) they are consistently slower. Do NOT enable on this hardware.

## Universal Base Flags (b9743, with overrides for tight VRAM)

`-ngl 99 --no-mmap --mlock --jinja --reasoning off --prio 2 --poll 30 --no-cont-batching --timeout 300 --threads 16 --threads-batch 32 --parallel 1 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --no-host -fitt 512 --temp 0.2 --top-p 0.95 --min-p 0.05 --top-k 20 --metrics`

**Add for tight VRAM builds:** `--no-warmup --batch-size 256 --ubatch-size 64` (swap `--no-warmup` for the warm-up run; reduce batch/ubatch to shrink compute graph buffers by ~1000 MiB).

## --n-cpu-moe Tuning for Speed

Lower `--n-cpu-moe` moves MoE expert weights from CPU to GPU, trading VRAM for speed.
Default of 128 puts all layer experts on CPU (safe, low VRAM). Reducing it puts later layers' experts on GPU.

**Sweep method to find optimal value:**
1. Test individual values: `--n-cpu-moe 44`, `--n-cpu-moe 42`, `--n-cpu-moe 40`, etc.
2. Check if it OOMs (error: unable to allocate CUDA0 buffer) or loads
3. Benchmark at the lowest value that loads

**Observed gains on 11GB RTX 2080 Ti:**

| Model | Before (ncm=128) | After | Gain | VRAM | Config |
|-------|-----------------|-------|------|------|--------|
| GLM-4.7-Flash-REAP-23B | 31 t/s | 54 t/s | **+72%** | 9.1 GB | ncm=20 (q4_0 cache) |
| GLM-4.7-Flash-REAP-23B (q8_0) | 31 t/s | 43 t/s | +38% | 9.8 GB | ncm=30 (q8_0 cache) |
| **GLM-4.7-Flash (Q4_K_M, 18GB)** | **25.6 t/s** | **36.2 t/s** | **+41%** | **10.3 GB** | **ncm=24 (best balance on 11GB)** |
| Nemotron-3-Nano-30B | 36 t/s | 47 t/s | +29% | ~9 GB | ncm=38 |
| Qwen3.6-35B-A3B-MTP | 51 t/s | 60 t/s | +17% | 9.2 GB | ncm=32 (decode only; prefill sweep below) |
| Qwen3-Coder-30B | 35 t/s | 38 t/s | +11% | 10.1 GB | ncm=42 |
| Qwen3-Coder-Next-80B | 33 t/s | 36 t/s | +8% | 8.4 GB | ncm=42 |
| **Ornith-1.0-35B-Q8** | — | **18 t/s** | — | 9.1 GB | **ncm=32** |
**Prefill-specific sweep: Qwen3.6-35B-A3B-MTP (Q4_K_M), ubatch=2048, --no-mmap --mlock:**

| n-cpu-moe | Prefill t/s | vs ncm=32 | VRAM Used | Free |
|:---:|:---:|:---:|:---:|:---:|
| 32 | **360** | baseline | ~8,900 MiB | ~2,100 MiB |
| 28 | **405** | +12% | 8,950 MiB | 2,053 MiB |
| 26 | **434** | +20% | 9,878 MiB | 1,125 MiB |
| 24 | **463** | +28% | 10,804 MiB | 198 MiB |

Decode speed is unaffected — these gains are prefill-only. ncm=26 is the sweet spot (+20%, safe headroom). ncm=24 works but leaves only ~200 MiB. These are faster than decode speeds because `--no-mmap --mlock` already gives pinned-memory H2D, and smaller ncm = fewer PCIe round-trips.

| **Ornith-1.0-35B-Q6-MTP** | — | **34 t/s** | **+78% vs Q8** | 9.9 GB | **ncm=30** |

**General guidance:**
- Smaller models (15-25 GB files) benefit most — moving a significant fraction of experts to GPU changes the CPU/GPU balance
- Larger MoE models (45+ GB files) gain less — even moving 6/48 layer experts barely changes the PCIe bottleneck
- A model at 2.6 GB VRAM with headroom is the best candidate for this optimization
- Stay above ~10.5 GB free VRAM to avoid OOM on compute buffers
- For dense models (no --n-cpu-moe flag), this doesn't apply
- Models with fewer layers than --n-cpu-moe value: all experts stay on CPU. e.g. --n-cpu-moe 128 on a 40-layer model = same as --n-cpu-moe 40 (all CPU)
- Sweep method: start from 128, drop by 5 until OOM, then back up 2-3 for safety margin

## Compute Graph Buffer Management

The **compute graph buffer** is often the true VRAM bottleneck, not model weights or KV cache. On 11GB, dense models at max context fail on `ggml_gallocr_reserve_n_impl: failed to allocate compute buffers` when weights + KV leave insufficient contiguous space.

### Symptoms
- Model loads and shows `model loaded` but fails during warm-up or first decode
- Error: `ggml_backend_cuda_buffer_type_alloc_buffer: allocating NNN MiB on device 0: cudaMalloc failed: out of memory`
- Error occurs at `graph_reserve` stage, not during model or KV allocation

### Mitigations (apply in order)

1. **Reduce `--batch-size` and `--ubatch-size`:** Halving batch roughly halves the compute buffer. Common reductions:
   | Batch | Buffer Size (Qwen3-Coder-Next-REAP-40B) |
   |-------|----------------------------------------|
   | 1024/256 | 1,632 MiB |
   | 512/128 | 488 MiB |
   | 256/64 | ~477 MiB |
   | 128/32 | ~477 MiB (diminishing returns below 256) |

   The trade-off: smaller batch reduces prompt processing throughput slightly but has near-zero effect on generation token-per-second speed.

2. **Use `--no-warmup`:** Skips the empty warm-up run that allocates a full-size compute buffer at startup. The buffer is allocated on first actual decode instead. Saves ~400 MiB of temporary VRAM during model load.

3. **Free up model VRAM:** Increase `--n-cpu-moe` (MoE models) to move more expert weights to CPU, leaving contiguous space for the compute buffer. Each layer of experts freed typically saves 200-450 MiB.

### Buffer size scaling
The compute buffer grows with context length for full-attention layers. At small contexts (~128K) it's ~477 MiB on Qwen3-Next architecture; at max native (262K) it grows to ~708 MiB. This is why models that load at 220K may OOM at 240K even though the KV cache difference is small.

<br>

After adding/removing models from llama-swap:
1. Update `~/.hermes/config.yaml`:
   - `custom_providers` → find `Local LLM (llama-swap)` → add/remove model entries
   - `fallback_providers` → add/remove fallback entries
   - `agent.environment_hint` → update model count
2. **Write caution:** Python `yaml.dump()` can corrupt config if args wrong. Safe call:
   ```python
   yaml.dump(cfg, f, default_flow_style=False, sort_keys=False, width=1000)
   ```
   Wrong: `default_flow_state` (typo) — silently creates empty file. Restore from `~/.hermes/state-snapshots/<latest>/config.yaml` if this happens.
3. **Restart gateway:** Old gateway process caches config at startup. Kill and restart:
   ```bash
   kill $(pgrep -f "hermes.*gateway.run") 2>/dev/null
   sleep 3
   /home/rurouni/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run
   ```
4. **Check Open WebUI** — models are stored in a hardcoded `model_ids` list in its SQLite DB, not auto-discovered from the API. Run `docker exec open-webui python3 -c "import sqlite3, json; conn = sqlite3.connect('/app/backend/data/webui.db'); row = conn.execute('SELECT id, data FROM config LIMIT 1').fetchone(); config = json.loads(row[1]); print(config['openai']['api_configs']['0']['model_ids'])"` to verify, then `docker restart open-webui`.

## Build Recipe (b9743+)

```bash
# CUDA 12.6 + g++-13 required (g++-14+ incompatible with CUDA 12.x nvcc)
cmake .. -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=75 \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-13 \
  -DCMAKE_C_COMPILER=/usr/bin/gcc-13 -DCMAKE_CXX_COMPILER=/usr/bin/g++-13 \
  -DCMAKE_BUILD_TYPE=Release
cmake --build . -j --config Release --target llama-server
# Install binary + shared libraries
sudo cp bin/llama-server /usr/local/bin/llama-server-sm75
sudo cp bin/*.so* /usr/local/lib/ 2>/dev/null
sudo ldconfig 2>/dev/null
```

**g++ version note:** CUDA 12.x nvcc supports up to g++-13. g++-14+ causes `__type_pack_element` errors in CUDA compiler ID checks.

**Binary sizes:** Dynamic linking -> ~18 KB binary + ~150 MB .so files.

## Per-Model Templates

**Active fleet (current deployment on 11GB RTX 2080 Ti — 5 models, Jul 3 2026):**\n\n| # | Model | Flags | Speed | VRAM | Ctx | Opt ncm | Role |\n|---|-------|-------|:----:|:----:|:---:|:------:|------|\n| 1 | **Qwen3.6-35B-A3B-MTP** (Q4_K_M) | `-c 230000 --batch 2048/512 --n-cpu-moe 26 --flash-attn on --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --temp 0.2 --top-p 0.95 --min-p 0.10 --top-k 20 --repeat-penalty 1.05 --repeat-last-n 256` | **51 t/s** | 9.9 GB | 230K | **26** (+20% prefill vs 32) | Primary daily coder |\n| 2 | **Gemma 4 26B A4B QAT** (Q4_K_XL) | `-c 230000 --batch 1024/256 --n-cpu-moe 16 --no-kv-offload --flash-attn on --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.85 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --model-draft <mtp.gguf> --temp 1.0 --top-p 0.95 --min-p 0.10 --top-k 40 --repeat-penalty 1.05 --repeat-last-n 256` | **52 t/s** | 10.5 GB | 230K | **16** | Deep reasoning |\n| 3 | **Ornith-1.0-35B-MTP-APEX** (I-Compact, 17 GB) | `-c 262144 --batch 2048/512 --n-cpu-moe 24 --no-kv-offload --flash-attn on --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --temp 0.6 --top-p 0.95 --min-p 0.10 --top-k 20 --repeat-penalty 1.05 --repeat-last-n 256`. APEX I-Compact (17 GB), MTP baked in, 262K context. **Lowest quality but fastest of Ornith-35B options.** | **28 t/s** | 9.9 GB | 262K | **24** | Self-improving agent. Alternatives: LordNeel IQ4_XS-MTP-graft (19.6 GB, better quality, KLD 0.0731) or Q5_K_M (24.7 GB, best fidelity KLD 0.0354). See `references/lordneel-ornith-35b-quality-ladder.md`. |\n| 4 | **Qwythos-9B-MTP-Q6** (Q6_K) **v3** | `-c 131072 --batch 1024/256 --flash-attn on --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --temp 0.2 --top-p 0.95 --min-p 0.10 --top-k 20 --repeat-penalty 1.05 --repeat-last-n 256`. **v3 fixes chat template looping**. 1M native ctx (YaRN). | **96 t/s** | 10.5 GB | 131K | n/a dense | Fastest reasoner |\n| 5 | **Ornith-1.0-9B-MTP-Q5** (Q5_K_M) | `-c 98304 --batch 2048/512 --flash-attn on --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --temp 0.6 --top-p 0.95 --min-p 0.10 --top-k 10 --repeat-penalty 1.10 --repeat-last-n 256`. **MTP baked-in GGUF** from protoLabsAI. | **71 t/s** | 8.9 GB | 96K | n/a dense | Fast dense coder |

**Previously benchmarked (models no longer in fleet — historical reference):**

| Model | Add to base | Speed | VRAM | Opt ncm |
|-------|------------|-------|------|---------|
| Qwen3-Coder-30B (qwen3moe) | `-c 131072 --batch 1024/256 --n-cpu-moe 42 --flash-attn on --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40` | 38 t/s | 10.1 GB | **42** |
| GLM-4.7-Flash-REAP-23B | `-c 131072 --batch 1024/256 --n-cpu-moe 30 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --reasoning off --temp 0.2 --top-k 40` | 43 t/s | 9.8 GB | **30** |
| Qwen3-Coder-Next-80B | `-c 131072 --batch 1024/256 --n-cpu-moe 42 --flash-attn on --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40` | 36 t/s | 8.4 GB | **42** |
| Nemotron-Term-14B (dense) | `-c 32768 --batch 1024/256 --flash-attn on --temp 0.2 --top-k 40` | 56 t/s | 10.6 GB | n/a |
| GPT-OSS-20B (MoE) | `-c 131072 --batch 1024/256 --no-kv-offload --flash-attn on --temp 0.7 --top-p 0.95 --top-k 40`. No --n-cpu-moe or -ngl 99 — let auto-fit handle offloading. | ~61 t/s | 9.9 GB | auto-fit |
| Ornith-1.0-35B-Q8 (Qwen3.5 MoE) | `-c 131072 --batch 2048/512 --n-cpu-moe 32 --no-kv-offload --flash-attn on --temp 0.6 --top-p 0.95 --min-p 0.05 --top-k 20`. 35 GB Q8_0 file. | ~18 t/s | 9.1 GB | **32** |
| Qwen-AgentWorld-35B-UD-Q6_K (A3B MoE) | `--jinja -c 131072 --batch 2048/512 --n-cpu-moe 28 --no-kv-offload --flash-attn on --temp 0.6 --top-p 0.95 --min-p 0.05 --top-k 20`. No `--reasoning off` — base/world model. | ~20 t/s | 10.3 GB | **28** |
| Gemma4-v2 agentic | `-c 131072 --batch 1024/256 --flash-attn on --temp 1.0 --top-p 0.95 --top-k 64`. No MTP. | 53 t/s | 7.7 GB | n/a |
| Nemotron-Nano-30B | `-c 131072 --batch 1024/256 --n-cpu-moe 38 --flash-attn on --temp 0.2 --top-k 40` | 47 t/s | ~9 GB | **38** |
| StrikeGPT-VL-8B Q6 | `-c 32768 --batch 1024/256 --flash-attn on --temp 0.2 --top-k 40`. LLM-only GGUF (vision tower stripped). | 70 t/s | 8.7 GB | n/a |
| WhiteRabbitNeo-13B Q4_K_M | `-c 16384 --ngl 28 --batch 1024/256 --flash-attn on --temp 0.2 --top-k 40`. Llama 2 arch, partial GPU fit. | 17 t/s | 10.2 GB | n/a |
| OpenAI-20B-NEOPlus Q8_0 | `-c 32768 --n-cpu-moe 32 --no-kv-offload --batch 1024/256 --flash-attn on --temp 0.2 --top-p 0.95 --top-k 40`. CPU-bound, abliteration degrades quality. | 18 t/s | 1.6 GB | CPU |

| Model | Add to base | Speed | VRAM | Opt ncm |
|-------|------------|-------|------|---------|
| Qwen3-Coder-30B (qwen3moe) | `-c 131072 --batch 1024/256 --n-cpu-moe 42 --flash-attn on --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40` | 38 t/s | 10.1 GB | **42** |
| GLM-4.7-Flash-REAP-23B | `-c 131072 --batch 1024/256 --n-cpu-moe 30 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --reasoning off --temp 0.2 --top-k 40` | 43 t/s | 9.8 GB | **30** |
| Qwen3-Coder-Next-80B | `-c 131072 --batch 1024/256 --n-cpu-moe 42 --flash-attn on --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40` | 36 t/s | 8.4 GB | **42** |
| Gemma4-v2 agentic | `-c 131072 --batch 1024/256 --flash-attn on --temp 1.0 --top-p 0.95 --top-k 64`. No MTP. | 53 t/s | 7.7 GB | n/a |
| Nemotron-Nano-30B | `-c 131072 --batch 1024/256 --n-cpu-moe 38 --flash-attn on --temp 0.2 --top-k 40` | 47 t/s | ~9 GB | **38** |
| Qwen3-30B-A3B-Thinking + Eagle3 | `-c 131072 --n-cpu-moe 128 --flash-attn on --spec-type draft-eagle3 --spec-draft-n-max 5 --spec-draft-p-min 0.75` | 36 t/s | 10.3 GB | 128 |

**Key notes on new additions:**

**GLM-4.7-Flash (non-REAP):** 30B/3B MoE, deepseek2 arch. 18.3 GB Q4_K_M file at 202K native ctx. Requires `--reasoning off` — GLM natively outputs `reasoning_content`. deepseek2 arch may need stripped flags (no --prio, --mlock, --cont-batching) if tensor_buft_overrides conflict arises, but b9743 handles standard flags for most deepseek2 models. Use `--no-kv-offload` to keep the 18.3 GB file's KV cache on CPU. Recommended Z.ai sampling: temp 1.0, top-p 0.95, min-p 0.01, repeat-penalty 1.0.

**GPT-OSS-20B:** 20.9B/3.6B active MoE (NOT dense!), 32 experts, 24 layers, gpt-oss arch. 12 GB Q6_K file at 131K native ctx. Native MXFP4 quantization in base model. Active params only 3.6B → very fast per-token when GPU-loaded. **CRITICAL: Do NOT use --n-cpu-moe or -ngl 99** — both disable auto-fit, and the 12 GB file OOMs with explicit offloading. Instead, use auto-fit + --no-kv-offload. This loads at ~9.9 GB VRAM (1.1 GB headroom) with 131K q8_0 KV cache on CPU. Higher temperature recommended (0.7) for creative output. HF source: unsloth/GPT-OSS-20B-GGUF. Uses Harmony chat format — reasoning goes to `reasoning_content`; final answer to `content`. Requires enough max_tokens for thinking overhead (~70 tokens for a simple answer).

**Context adjustment rule:** Increasing context while using `--no-kv-offload` has ZERO VRAM impact — KV cache stays on CPU. Only affects prompt processing throughput.

**30B-Thinking note:** Base model for Eagle3 speculator. Once speculator converted and paired: ~40-50+ t/s with `--spec-type draft-eagle3`.

## CDN Speed Differences (HF downloads)
- **unsloth**: ~75-100 MB/s
- **bartowski/mradermacher**: ~50-80 MB/s  
- **lovedheart** (small accounts): ~9 MB/s (very slow)
Use `notify_on_complete=true` with long timeout for slow CDNs. Do NOT kill/retry — resume works.

**Coder model flags (from unsloth guide):** Qwen3-Coder and Coder-Next models use higher temperature and wider sampling than general chat models: `--temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40`. These models are non-thinking (`--reasoning off`). All Qwen3-Coder variants (30B, Next, Next-REAP) use MoE via `--n-cpu-moe 128`.

**Known gemma-12b regression:** 96 vs 118 t/s — upstream CUDA bugs #24514, #24670 on Turing. Not fixable from our side.

## Frankenstein MTP Grafting (Cross-Model MTP Heads)

When a model doesn't ship with native MTP heads, they can be **surgically grafted** from a same-architecture sibling that does have MTP. This enables `--spec-type draft-mtp` self-speculative decoding on models that never trained with MTP.

**How it works:** 20 MTP head tensors (attention q/k/v/output norms, shared + routed experts gate/up/down, nextn projection heads) are extracted from the donor model and appended to the target GGUF. The metadata is patched: `block_count` +1, `nextn_predict_layers` = 1.

**Prerequisites:**
- Donor and target must share the same architecture (`qwen35moe`, `gemma4`, etc.), embedding dimensions, and layer structure
- The graft script (`gguf_mtp_graft.py`) and donor heads are published by **skinnyctax** on HuggingFace
- llama.cpp build must support `--spec-type draft-mtp`

**Usage:**
```bash
# 1. Download donor heads + graft script
hf download <user>/<repo> <donor-heads.gguf> gguf_mtp_graft.py --local-dir .

# 2. Graft onto your base GGUF (match donor quant to target quant)
python3 gguf_mtp_graft.py your-base-Q6_K.gguf donor-heads-Q6_K.gguf output-MTP.gguf

# 3. Run with self-speculation
llama-server --model output-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 ...
```

**Quant matching rule:** Match donor quant to target quant for best acceptance. Q6 heads on Q6 base, Q4 heads on Q4 base. Cross-quant grafting works but has lower acceptance.

**Known working sources:**
- skinnyctax/Ornith-1.0-35B-Q6_K-Frankenstein-MTP-GGUF (Ornith Q6/Q4 variants + DIY heads)
- wang-yang/Ornith-1.0-35B-MTP-GGUF (Q6, `--spec-draft-n-max 1` found best on M3 Max)
- s-batman/Ornith-1.0-35B-NVFP4-MTP-GGUF (Blackwell-only, sm_120+)

**Performance typical:** ~1.2-1.3× speedup over non-MTP baseline. MTP acceptance rate 74-95% depending on quant. Q4 acceptance often *higher* than Q6 due to noise alignment with draft predictions.

**Limitations:**
- Not all architectures support grafting — only those where a same-arch MTP donor exists
- RL-trained finetunes may have more weight drift from the donor, reducing acceptance
- The MTP head costs ~280 MiB KV context + ~0.5-0.7 GB VRAM for the head weights
- Requires llama.cpp b9743+ with `draft-mtp` support

## Research Workflow (mandatory)
1. **Always web-research before guessing flags.** Check HF model card → unsloth/docs → GitHub issues/PRs. Trial-and-error without research wastes time.
2. **Negative claims require evidence.** Before asserting a model, format, or quantization "doesn't exist," search concretely. "Doesn't exist" is a high-confidence claim — verify it, don't assume it. Common failure: assuming GGUF format is unavailable (e.g. DSpark speculators now have GGUF quantizations for Qwen3-8B, Gemma4-12B — look on HF before saying no GGUF exists). If you can't find it, say "not found in search" not "doesn't exist."
3. **For MoE models: file size ≠ VRAM.** A 25 GB Q5_K_M MoE file does NOT need 25 GB of VRAM. With `--n-cpu-moe`, only non-expert layers (~2-3 GB for a 35B MoE) live on GPU. Quantization level affects PCIe bandwidth (speed), not fit. Never say "Q5_K_M won't fit in 11 GB" — say "Q5_K_M will be slower than Q4 by ~X% due to more data across PCIe."
4. If stuck on flag compatibility or performance regression, search GitHub issues (not just help output).
5. Test directly (not through llama-swap) with minimal flags first.
6. For new models, check arch from GGUF metadata via `timeout 8 llama-server --model file.gguf --no-mmap -c 512 2>&1 | grep -E "arch|type |file type"`.
7. For large GGUF files (>5 GB) where llama-server metadata times out, use Python raw-byte parsing (see `references/moe-research-and-gguf-parsing-2026-06-22.md` for v3 header format).
8. When researching MoE offloading for a new model on limited VRAM: load model card → estimate ncm using file_size/layer method → test with `--no-kv-offload` if context is large → sweep ncm down from 128 by increments of 5.

## Key Pitfalls
- Prefer web research over trial-and-error when stuck on flag compatibility
- Diffusion models (DiffusionGemma) need special PR branch + dedicated CLI — no HTTP server exists
- **`--n-cpu-moe` + `-ngl 99` + `--mlock` can break auto-fit on large GGUF files.** When a model's GGUF file is close to or larger than VRAM (e.g. 12 GB Q6_K on 11 GB card), setting both `-ngl 99` and `--n-cpu-moe` triggers `tensor_buft_overrides` which disables auto-fit. Result: `cudaMalloc failed: out of memory` at model load. **Fix:** Remove both `-ngl` and `--n-cpu-moe` — let auto-fit handle offloading. Add `--no-kv-offload` to free VRAM for model weights. Verified on GPT-OSS-20B (12 GB Q6_K): auto-fit + --no-kv-offload loads at 9.9 GB; -ngl 99 + --n-cpu-moe 10 OOMs at 6.8 GB allocation.
- DeepSeek v2/v3 arch: strip flags to minimum
- **`--reasoning off` behavior varies by architecture:**
  - GLM models: fully obeys — without it, GLM outputs empty `content` and all text in `reasoning_content`
  - Qwen models (qwen36, qwen3-coder-*): still emit `<think>`/`<thinking>` tags regardless of flag
  - Gemma, Nemotron: cleanly suppressed, no residual tags
  - Always test the target model after adding/removing this flag
- **llama-swap `sendLoadingState: true` + model crash = confusing "thinking" output.** When a model server crashes during load (e.g. CUDA OOM), llama-swap emits a loading SSE event that Open WebUI renders as the model's thinking/reasoning output. The user sees `llama-swap loading model: X` instead of an error message. Always check llama-swap logs for `exited prematurely` entries when a model produces no real response.
- **Open WebUI model routing issues** — diagnose via SQLite DB inspection. See `references/open-webui-model-routing-diagnosis.md` for step-by-step.
- **Cherry-picking patches onto wrong base = model load failure.** If building from a stale commit (e.g. 048a490 behind c576070), SSM support for qwen35moe is missing and models fail with `missing tensor 'blk.40.ssm_conv1d.weight'`. Always verify base commit matches the working binary (`llama-server-sm75 --version`).
- **`rm -rf build` is mandatory after `git reset --hard`.** CMake doesn't detect changed CUDA template instantiations across different base commits. Incremental builds produce undefined-reference linker errors (`ggml_cuda_flash_attn_ext_*`). Full clean rebuild required.
- **`GGML_SCHED_PREFETCH_EXPERTS=1` is counter-productive on RTX 2080 Ti.** The MoE prefetch patches (see `references/fable5-moe-prefetch-patches.md`) add CUDA stream overhead that outweighs any overlap benefit on Turing + PCIe Gen3 + `--no-mmap` (already pinned). Leave unset. See reference for benchmark data.
- **`GGML_CUDA_GRAPH_OPT=1` and `GGML_CUDA_FORCE_CUBLAS_COMPUTE_16F=1` have zero measurable effect on this hardware.** Tested Jul 2026 on Qwen3.6-35B prefill — no speed change from either env var alone or combined. Don't bother retesting.

## Adding Models to Hermes

After adding a profile to llama-swap, update `~/.hermes/config.yaml` under `custom_providers` and `fallback_providers`. See `references/hermes-config-update.md` for the exact format.