# Verified Benchmarks — 2080 Ti (2025-05-26)

## Methodology
- Binary: `/usr/local/bin/llama-server-sm75` (llama.cpp b9341, sm_75, CUDA 12.6)
- Test: `curl` POST to `/v1/chat/completions`, prompt "Write a Python quicksort with comments. Return ONLY code.", max_tokens=256, temp=0.2
- All tests with: turbo3 K, turbo4 V, flash-attn on, --no-mmap, --mlock, --no-host, --cont-batching
- Metric: completion_tokens / wall-clock time (includes HTTP/json overhead — real-world)

## Optimal Flags Per Model

### 35B MoE (heretic-35b, qwen36-35b-mtp)
```
--spec-type mtp --spec-draft-n-max 4
--spec-draft-type-k q4_0 --spec-draft-type-v q4_0
--threads 16 --threads-batch 32
--batch-size 2048 --ubatch-size 512
--n-cpu-moe 128
```

### 9B Dense (qwopus-9b, ds-flash-9b, qwen35-9b-mtp)
```
--spec-type mtp --spec-draft-n-max 4
--spec-draft-type-k q4_0 --spec-draft-type-v q4_0
--threads 16 --threads-batch 32
--batch-size 1024 --ubatch-size 256
--no-kv-offload
```

## Full Fleet Results

| Model | Model ID | tok/s | Context | VRAM | Load |
|---|---|---|---|---|---|
| Heretic 35B MTP | heretic-35b | **48.8** | 250K | 5,064 MiB | 22s |
| Qwen3.6 35B MTP unsloth | qwen36-35b-mtp | **48.6** | 200K | 5,144 MiB | 58s |
| Qwopus 9B Coder MTP | qwopus-9b | **45.8** | 150K | 8,330 MiB | 19s |
| Qwen3.5 9B MTP UD | qwen35-9b-mtp | **45.7** | 200K | 7,182 MiB | 6s |
| DeepSeek Flash 9B MTP | ds-flash-9b | **33.0** | 200K | 8,566 MiB | 19s |
| Gemma 4 26B A4B | gemma-26b | **30.1** | 131K | 3,868 MiB | 47s |
| Gemma 4 12B Dense | gemma-12b | **53.6** | 131K | 7,300 MiB | -s |

## Gemma 4 12B Optimal Flags

No MTP, native thinking (outputs reasoning_content before content).

```
--jinja
-c 131072 --ctx-size 131072
--batch-size 1024 --ubatch-size 256
--cache-type-k turbo3 --cache-type-v turbo4
--flash-attn on --no-host -fitt 512
--threads 16 --threads-batch 32 --parallel 1
--temp 0.2 --top-p 0.95 --min-p 0.05 --top-k 20
--metrics
```

**Key differences from 9B dense:**
- No MTP. `gemma4` arch has no MTP head support in llama.cpp (build b9341). PR #24057 (E4B MTP) was closed, never merged.
- No `--reasoning off` by default (add it to suppress native thinking trace).
- No `--no-kv-offload` — 7.3 GiB fits entirely in 11GB.
- Context: 131K native. Do NOT set 200K.
- Prompt processing: 348 t/s.

## Flag Comparison on heretic-35b @ 250K

| draft | threads | batch/ubatch | tok/s | VRAM |
|---|---|---|---|---|
| 4 | 16 | 2048/512 | **48.8** | 5.0GB |
| 4 | 32 | 2048/512 | 37.0 | 5.0GB |
| 3 | 32 | 256/256 | 38.7 | 4.2GB |
| 3 | 16 | 256/256 | 37.0 | 4.2GB |
| 2 | 32 | 256/256 | 37.8 | 4.2GB |
| 2 | 16 | 256/256 | 36.5 | 8.6GB |

**Conclusion:** draft=4, threads=16, batch=2048/512 is the clear winner. Any change costs 10-12 t/s.

## Regression Event (2025-05-26)
A config update replaced draft=4→2, threads=16→32, batch=2048/512→256/256 based on incorrect assumptions from 9B dense model testing. This dropped heretic-35b from 48.8 → 36.6 t/s. Root cause: the llama-server-flags skill had stale/incorrect MTP and batch-size recommendations that did not match the user's production benchmarks. All corrected.

## CRITICAL: Never extrapolate flags across model classes
When testing a new model (e.g. qwen36-35b-mtp, unsloth variant) and finding optimal flags, do NOT apply those flags to other models — even ones sharing the same architecture family (e.g. heretic-35b). Each model/quant combination needs independent verification.

The correct workflow:
1. Test the NEW model with its own variants
2. Test the EXISTING models with their OWN known-good flags (do not change them)
3. Only change an existing model's flags if you A/B test it on THAT model

The regression occurred because draft=2 + batch=256/256 was optimal for qwen36-35b-mtp (new unsloth model) but terrible for heretic-35b (production model). The lesson: **always consult bench-2080ti.sh first; it is the authoritative source for production model flags.**