---
name: llm-benchmark-optimization
description: Methodology for benchmarking and optimizing llama.cpp models on constrained GPUs — flag verification, benchmark protocols, drift detection, and research-validation discipline.
---

# LLM Benchmark & Optimization Methodology

## When to Use

- Tuning llama-server flags for a new model or GPU
- A/B testing flag variations (MTP draft length, batch size, cache types)
- Auditing a config after flag drift (e.g., "did someone change the Qwen flags?")
- Verifying research agent recommendations against actual binary help output
- Running fleet-wide optimization passes

## Core Principles

### 0. When Stuck — Research First, Guess Never

**User correction received:** "do web research look at llama cpp look for updated server you need to stop guessing and creating things always web research when you get stuck"

When facing an unknown architecture, flag behavior, or model capability:
1. **Read the model card** on Hugging Face — architecture, context length, recommended sampling params. Read raw README.md: `curl -sL "https://huggingface.co/<org>/<model>/raw/main/README.md" | head -200`
2. **Check the GitHub repo** — recent PRs, closed issues, release notes. Use API: `curl -s "https://api.github.com/repos/ggml-org/llama.cpp/compare/<OLD_HASH>...master" | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'{d.get(\"total_commits\",\"?\")} commits, {d.get(\"behind_by\",\"?\")} behind')"`
3. **Search specific PRs by label** — `curl -s "https://api.github.com/search/issues?q=repo:ggml-org/llama.cpp+type:pr+is:merged+label:cuda+updated:><DATE>&per_page=10" | python3 -c "..."` — filter by cuda, server, spec labels
4. **Check release notes** — `curl -s "https://api.github.com/repos/ggml-org/llama.cpp/releases?per_page=15" | python3 -c "import sys,json,re; ..."` — strip HTML tags, filter for meaningful commit messages
5. **Look for official docs** — Unsloth guides, Google DeepMind docs, PR descriptions
6. **Only then test** — with minimal flags, direct (not through llama-swap)

Do NOT:
- Guess flag names or values without verification
- Try random combinations hoping something works
- Assume a feature exists because it sounds plausible
- Build custom binaries without checking if the feature already exists in upstream master

### 1. Never Trust — Always Verify Against `--help`

Research agents (delegate_task) **consistently hallucinate flags**. Verified examples:
- `--cuda-streams 2` — does not exist on llama-server-sm75 OR upstream binary
- `--spec-type draft-mtp` recommended for Qwen — sm75 binary only accepts `mtp`
- Poll described as "milliseconds" — actual: level 0-100

**Always run** `/usr/local/bin/llama-server-<build> --help 2>&1 | grep <flag-name>` before applying a new flag.

### 2. Binary Version Matters — Track `--version` Output

Every llama.cpp build has a unique version string. Record it explicitly:

```bash
/usr/local/bin/llama-server-sm75 --version 2>&1 | head -1
# → "version: 9743 (c57607016)"
```

**When binaries get rebuilt from the same source** (happened June 2026: both sm75 and upstream now from master b9743), flag vocabularies converge:

| Old (b9341/ac4cdde) | New (b9743) | Impact |
|---------------------|-------------|--------|
| `--spec-type mtp` (sm75 only) | `--spec-type draft-mtp` (unified) | MUST update — old `mtp` crashes on new binary |
| Turbo KV cache (sm75 only) | No turbo in upstream master | q8_0 is floor |
| 2 separate CUDA code paths | 1 unified CUDA path | Some models faster, some slower |

**Performance regression/recovery pattern observed:**
- Qwen36 gained +12% (45.8 → 51.4 t/s) after rebuild
- Gemma-12b lost -18% (117.8 → 96.2 t/s) — different CUDA kernel selection in new binary
- Both at exactly same flags — the compiler/CUDA version difference is the only variable

Always re-benchmark the top-3 most-used models after any binary rebuild. Record: t/s, VRAM, cold start time, draft acceptance stats.

### 3. Benchmark Protocol (Single Model, Constrained GPU 11GB)

**Setup:**
```bash
# Save payload to file (avoid shell JSON escaping issues)
echo '{"model":"<id>","messages":[{"role":"user","content":"Write a Python quicksort with comments. Return ONLY code."}],"max_tokens":256,"temperature":0.2,"stream":false}' > /tmp/bench_payload.json

# Run benchmark — capture response body (-o) AND wall clock (-w) in one call
curl -s --max-time 300 -X POST http://127.0.0.1:9292/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d @/tmp/bench_payload.json \
  -o /tmp/bench_response.json \
  -w "HTTP_CODE:%{http_code}\nTIME_TOTAL:%{time_total}\n"
```

**Key rules:**
- **Cold load skew** — llama-swap keeps models in VRAM for TTL (3600s). A 0.15s "load" means it was cached, not cold. Force-unload by loading a different model or restart llama-swap between tests.
- **3-run averaging for MTP** — MTP draft acceptance varies by prompt (observed range: 44.6-49.9 t/s on Qwen 35B). Individual runs are noisy. Report median or 3-run average.
- **Use `timings` dict, not wall clock alone** — llama.cpp returns `predicted_per_second`, `predicted_per_token_ms`, and `draft_n_accepted` in the response. These are cleaner than wall-clock timing for generation speed. Use `-w` wall clock for cold-load latency and broad timing context.
- **VRAM before/after** — Record `nvidia-smi --query-gpu=memory.used` before and after loading. The delta is the model's real footprint. Background GPU processes add noise.
- **Record finish_reason** — "length" = hit max_tokens. "stop" = model stopped early (may indicate issues).

### 3b. Fleet-Wide Benchmark Protocol (All Models Sequentially)

When benchmarking all models sequentially (after rebuild, config change, or new addition):

**Orchestration pattern:** Drive the loop with Python — generate payloads with `json.dumps()`, pass to curl with proper shell quoting, parse both `-w` wall clock and response JSON `timings` for each run.

```python
def make_payload(model_id, prompt_text, max_tokens):
    return json.dumps({
        "model": model_id,
        "messages": [{"role": "user", "content": prompt_text}],
        "max_tokens": max_tokens, "temperature": 0.2, "stream": False
    })

def shq(s):
    """Minimal shell-quoting for JSON string — works robustly."""
    return "'" + s.replace("'", "'\\''") + "'"

def curl_bench(payload_json, timeout=120):
    cmd = (
        f'curl -s --max-time {timeout} -X POST http://127.0.0.1:9292/v1/chat/completions '
        f'-H "Content-Type: application/json" -d {shq(payload_json)} '
        f'-o /tmp/bench_out.json -w "HTTP_CODE:%{{http_code}}\\nTIME_TOTAL:%{{time_total}}\\n"'
    )
    subprocess.run(cmd, shell=True, ...)
    # Parse /tmp/bench_out.json for timings dict
    # Parse stdout for wall clock (TIME_TOTAL)
    # Record nvidia-smi for VRAM
    # Return dict with gen_tps, prompt_tps, wall_time, vram, draft stats
```

**Key rules for fleet sweeps:**
- **Cold vs warm separation** — run 1 for each model includes model load time (30-60s). Skip for speed reporting; record separately as cold-load latency.
- **3 benchmarks per model**: generation speed (512-token output), prompt processing (~3K-token input), stress test (5 rapid short requests).
- **Always save raw results** — write all runs to a JSON file for post-hoc analysis. Don't rely on inline print only.
- **Pipeline order** — benchmark models smallest-first so earlier runs don't waste VRAM on large models.
- **Total time for 5-model fleet**: ~10-15 minutes (allowing for cold loads and swap overhead).
- **Do NOT drive fleet sweeps via execute_code** — the 300s execute_code timeout kills multi-model tests that take 10-15 minutes. Use the standalone `scripts/fleet-bench.py` script run via `terminal(background=true, notify_on_complete=true)` instead. Poll progress with `process(action='poll')` or `terminal('wc -l /tmp/bench_results.json')` to check completion.
- **Reusable script**: see `scripts/fleet-bench.py` in this skill directory — run it directly with `python3 scripts/fleet-bench.py`.
- **`n-cpu-moe sweep script`**: `scripts/ncm-sweep.sh` — standalone bash script for testing a range of `--n-cpu-moe` values on a single model. Edit the variables at the top and run. Outputs VRAM + TG speed per value. Tested on Ornith-1.0-35B-Q8, adaptable to any MoE model.

### 3c. Fleet Stress Test Protocol (Sustained Rapid Requests)

Tests model stability under rapid-fire sustained load — distinct from context stress testing (max context boundaries).

**Protocol (per model):**
1. After model is loaded and warm (post-gen or post-prompt benchmark), fire 5 rapid sequential requests
2. Use short prompts (25-30 tokens) with max_tokens=50 — fast completions minimize cooldown between requests
3. Measure: first-request wall time (cold-load residue), subsequent average wall time, min wall time, generation t/s variance

**Interpretation:**
- **First time >> average** → expected cold-load latency. Report as cold-start time separately.
- **Min time** → true response latency when model is fully loaded and cached (typ. 0.4-0.9s for small prompts on 11GB)
- **Gen t/s variance > 10% between warm requests** → possible thermal throttling, VRAM contention, or process priority issues
- **Draft/acceptance rate on stress requests** — if MTP acceptance drops below 80% under rapid fire, draft may have a cooldown artifact

**Expected on RTX 2080 Ti 11GB (validated June 2026):**
| Model | Cold load | Warm stable t/s | Min response |
|-------|-----------|-----------------|-------------|
| nemotron-term-14b | 18-28s | 59-60 | 0.40s |
| gpt-oss-20b | 33-38s | 65-67 | 0.89s |
| glm-4.7-flash (ncm=24) | 48-67s | 35-36 | 0.76s |
| qwen36-35b-mtp | 58-66s | 76-78 | 0.36s |
| gemma-26b-200k | 37-44s | 86-88 | 0.42s |

### 4. Flag Verification Checklist (After Any Config Change)

```bash
grep -c "chat-template-kwargs" ~/.config/llama-swap/config.yaml      # must be 0
grep -c "reasoning off" ~/.config/llama-swap/config.yaml              # must equal model count
grep -c "spec-draft-type-k" ~/.config/llama-swap/config.yaml          # must cover all MTP/draft-mtp models
grep "min-p 0 " ~/.config/llama-swap/config.yaml                      # must be empty (min-p=0 is broken)
grep -c "no-host -fitt" ~/.config/llama-swap/config.yaml              # should match model count
```

### 5. Context Stress Testing Protocol (Max Context Determination)

When setting a model's context length, empirically determine the maximum by progressively testing:

**Protocol:**
```bash
# 1. Start with conservative context (1K) to confirm service loads
# 2. Double until crash, then binary search midpoint
# 3. If q8_0 KV fails at target context, try q4_0 KV
# 4. Record the working max for each KV quant
```

**Sequence:** 1K (sanity) → 128K → 192K → 256K → if all pass, you have headroom

**Interpretation when q8_0 fails but q4_0 works:**
- The model's KV cache is the primary VRAM consumer at high context
- q4_0 halves KV cache VRAM at ~0.5% quality cost (imperceptible for most tasks)
- The delta between q8_0 max and q4_0 max is exactly 2× (since q4_0 = 2 tokens per byte vs q8_0 = 1)
- If even q4_0 fails at native context, the model won't reach its advertised context on this GPU

**Known results (RTX 2080 Ti 11GB):**
| Model | File | q8_0 Max | q4_0 Max | Notes |
|-------|------|----------|----------|-------|
| Qwen3.6 FableVibes 14B MoE | 7.9 GB | ~192K | **256K (native)** | Prompt 1500+ t/s, Gen 64-106 t/s |
| Qwen3-Coder-Next 80B | 46 GB | **140K** (ncm=46) | Untested | 33.7 t/s validated |

**Current fleet benchmark (June 2026, 512-token gen, ~3.2K-token prompt processing):**
| Model | File | VRAM | Gen t/s | Prompt t/s | Cold load | Notes |
|-------|------|------|---------|------------|-----------|-------|
| nemotron-term-14b | Q4_K_S (8GB) | 10.8 GB | 56.2 | 1,292 | 18-28s | Fast prompt ingestion, dense 14B |
| gpt-oss-20b | Q6_K (12GB) | 9.9 GB | 60.9 | 595 | 33-38s | Lightest MoE, low VRAM footprint |
| glm-4.7-flash | Q4_K_M (18GB) | 10.3 GB | 36.2 | 87 | 48-67s | n-cpu-moe 24 (+41% over stock ncm=40), KV in sys RAM |
| qwen36-35b-mtp | Q4_K_M (22GB) | 10.9 GB | 66.3 | 248 | 58-66s | MTP 97.5% accept, maxed VRAM |
| gemma-26b-200k | Q4_K_XL (14GB) | 10.5 GB | 66.1 | 251 | 37-44s | +MTP draft 90% accept, good balance |

### 5. Scale-Back Priority (When OOM or Unstable on 11GB)

Do NOT start by scaling KV cache. This order preserves quality longest:

1. **Gemma 12B context** — 256K → 131K (saves ~1.6GB VRAM, same speed)
2. **MTP tuning** — reduce `--spec-draft-n-max` or increase `--spec-draft-p-min`
3. **Safety margin** — `-fitt 512` → `768` → `1024`
4. **Batch size** — `2048` → `1024` if prompt ingestion unstable
5. **Ubatch** — `512` → `256` if CUDA memory unstable
6. **MoE CPU placement** — adjust `--n-cpu-moe`
7. **KV cache** — only as last resort

#### 7. --n-cpu-moe Sweep for MoE Models

When optimizing MoE models with CPU expert offload (`--n-cpu-moe`), systematically find the lowest safe value:

**Protocol — Sequential (recommended for single-model sweeps):**
```bash
# 1. Start server as background process: terminal(background=true, command="... --n-cpu-moe $NCM")
# 2. Wait for health: curl health endpoint in a loop (10s timeout between retries, up to 60s)
# 3. Run gen bench (2 runs of 256 tokens) via curl, parse timings dict from response
# 4. Record VRAM (nvidia-smi), gen t/s, prompt t/s
# 5. Kill server, wait 2s for VRAM release, repeat with next ncm value
```

**IMPORTANT:** Do NOT use `execute_code` for n-cpu-moe sweeps — the 300s timeout kills multi-model tests. Use `terminal(background=true)` + health-check poll loop instead. For the same reason, avoid inline bash `for` loops with `&` — the Hermes terminal tool blocks `&` syntax.

**Interpretation:**
- The goal is the lowest `--n-cpu-moe` that loads without OOM
- Lower = more experts on GPU = faster generation
- The speed improvement is NOT linear — look for threshold points where a critical batch of expert layers fits in VRAM
- Start from the current ncm value and decrement by 4-8 at a time. Binary search the boundary once OOM is found
- `--no-kv-offload` keeps KV cache in system RAM, so context length does not affect VRAM during an ncm sweep — this simplifies diagnosis
- Sweep the model's actual binary directly (not through llama-swap) to isolate the flag effect from swap latency

**Known results (RTX 2080 Ti 11GB):**
| Model | File | ncm | Cache | VRAM | Gen t/s | Prompt t/s | Δ from stock |
|-------|------|-----|-------|------|---------|------------|-------------|
| GLM-4.7-Flash | Q4_K_M (18GB) | 40 | q8_0 (sys RAM) | 4.9 GB | 25.6 | 26 | — (stock) |
| GLM-4.7-Flash | Q4_K_M (18GB) | 32 | q8_0 (sys RAM) | 7.6 GB | 29.3 | 85 | +14% gen, +227% prompt |
| GLM-4.7-Flash | Q4_K_M (18GB) | 28 | q8_0 (sys RAM) | 9.0 GB | 28.8 | 86 | ~same as 32 |
| GLM-4.7-Flash | Q4_K_M (18GB) | 24 | q8_0 (sys RAM) | 10.3 GB | 36.2 | 87 | **+41% gen, +235% prompt** |
| GLM-4.7-Flash | Q4_K_M (18GB) | 20 | — | OOM | — | — | Crashes on load |
| Nemotron-3-Nano-30B | 23 GB | 38 | q8_0/q8_0 | ~9 GB | 46.8 | — | +29% |
| Coder-Next-80B | 45 GB | 42 | q8_0/q8_0 | 8.4 GB | 35.7 | — | +8% |

> Full sweep data with VRAM per step and test methodology: `references/glm-4.7-flash-ncm-sweep.md`
> AgentWorld-35B-UD-Q6_K sweep: `references/qwen-agentworld-35b-ncm-sweep.md`
> Ornith-35B-Q8_0 sweep: `references/ornith-35b-q8-ncm-sweep.md`

### 8. Eagle3 Speculator Limitations

Eagle3 speculative decoding requires:
1. Exact matching base model (architecture-locked)
2. Converted GGUF (safetensors -> GGUF via convert_hf_to_gguf.py)
3. Base model config files (tokenizer.json, config.json) for conversion via --target-model-dir

**Performance note:** Eagle3 provides minimal speedup on CPU-offloaded MoE models (< +5%). The bottleneck is PCIe bandwidth for expert weight transfer, not token prediction speed. Eagle3 gains are only realized on GPU-bound configurations (all layers on GPU, no CPU offload).

- **Inline JSON curl breaks** — shell variable expansion and quoting issues with `-d '{"key":"value"}'`. Two safe approaches: (a) always use `-d @/tmp/file.json`, or (b) when generating payloads programmatically, use Python's `subprocess.run` with `json.dumps()` + the `shq()` shell-quoting technique (`"'" + s.replace("'", "'\\''") + "'"`) — this handles single-quotes, newlines, and special characters reliably.
- **Piping curl to python parses headers** — use `-o /tmp/response.json` then parse separately.
- **Second-resolution timing** — for speeds >30 tok/s, second-resolution in wall clock gives 0-second durations. Use the metric timestamps from `timings` dict.
- **MTP variability on Qwen** — observed 44.6-49.9 t/s range on identical prompts. This is normal MTP behavior (draft acceptance varies).
- **Research agents invent flags** — `--cuda-streams 2` was recommended by deep research and does not exist on any llama.cpp build. Always verify against `--help`.
- **API endpoints may cache model lists in hardcoded DB entries.** Open WebUI stores model IDs in a `config` table in its SQLite DB, not auto-discovered from the API. After adding/removing models: `docker exec open-webui python3 -c "import sqlite3, json; conn = sqlite3.connect('/app/backend/data/webui.db'); row = conn.execute('SELECT id, data FROM config LIMIT 1').fetchone(); config = json.loads(row[1]); config['openai']['api_configs']['0']['model_ids'] = [...]"` then `docker restart open-webui`.
- **llama-swap `sendLoadingState: true` + model crash = confusing thinking output.** When a model server crashes during load, llama-swap emits a loading SSE event that Open WebUI renders as the model's thinking output. The user sees `llama-swap loading model: X` instead of an error. Always check llama-swap logs for `exited prematurely` when a model produces no real response.
- **`--reasoning off` behavior is model-architecture-dependent.** It fully suppresses output for GLM models (forces `reasoning_content` into `content`), but Qwen models (qwen36, qwen3-coder-next) still emit `<think>`/`<thinking>` tags regardless. Gemma and Nemotron models obey `--reasoning off` with no residual tags. Always test the target model directly after setting this flag.

## Upstream Regression Hunting

When a benchmark shows a drop after a rebuild, do **not** assume config error. Check for known upstream regressions:

1. **Find the bad commit range** — compare old vs new build numbers:
   ```bash
   curl -s "https://api.github.com/repos/ggml-org/llama.cpp/compare/b9341...master" | \
     python3 -c "import sys,json; d=json.load(sys.stdin); print(f'{d.get(\"total_commits\",\"?\")} commits')"
   ```

2. **Search for relevant issues** by label + date:
   ```bash
   curl -s "https://api.github.com/search/issues?q=repo:ggml-org/llama.cpp+type:pr+is:merged+spec+in:title+updated:><DATE>&per_page=10" | \
     python3 -c "import sys,json; d=json.load(sys.stdin); print(f'{d[\"total_count\"]} PRs'); [print(f'  #{i[\"number\"]}: {i[\"title\"][:100]}') for i in d.get('items',[])]"
   ```

3. **Check for architecture-specific issues** — Turing (sm_75) specific bugs exist (e.g. #24670: draft-mtp not activating on Turing). Search by GPU arch:
   ```bash
   curl -s "https://api.github.com/search/issues?q=repo:ggml-org/llama.cpp+Turing+sm_75+in:title&per_page=5"
   ```

4. **Check the first-bad-commit pattern** — issues often report which build introduced the regression. Cross-reference with your old build to see if you're past the regression point.

5. **If issue is confirmed open** (not fixed), record it in the model's config comments. No point wasting time on workarounds.