---
name: llama-cpp
description: Battle-tested llama-server flags for RTX 2080 Ti (sm_75, Turing, 11GB) with MTP + TurboQuant. Production templates for MoE, non-MoE, Gemma.
---

# llama-server Flag Templates

**Target:** RTX 2080 Ti 11GB, CUDA sm_75 (Turing), driver 550.163.01, CUDA 12.6

## Build Selection
| Build | Binary | Use for |
|-------|--------|---------|
| sm75 (b9743) | `llama-server-sm75` | **All models** — unified build from upstream master. Single binary for everything. |
| upstream (ac4cdde) | `llama-server-upstream` | **Retired** — replaced by b9743 unified build. Old backups removed. |

**Important:** The old binary split (mtp-turbo fork b9341 vs upstream ac4cdde) is obsolete.
The unified b9743 build replaces both. TurboQuant KV cache types (turbo3/turbo4) are NOT
available in upstream master — use q8_0 cache instead.
MMVQ_TURING optimization added in b9743 — Turing-specific mat-vec parameters for RTX 2080 Ti.

## Known CUDA Regressions (b9743 vs old builds)
| Issue | Impact | Upstream Status |
|-------|--------|----------------|
| #24514 | ~25% gen speed drop b9301→b9305 affecting MTP spec decoding. gemma-12b dropped from 117.8→96.2 t/s | Open, not fixed |
| #24670 | `draft-mtp` not activating on Turing sm_75 with hybrid SSM+attention models | Open, not fixed |

## Universal Base Flags (b9743, sm_75)
`-ngl 99 --no-mmap --mlock --jinja --reasoning off --prio 2 --poll 30 --no-cont-batching --timeout 300 --threads 16 --threads-batch 32 --parallel 1 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --no-host -fitt 1024 --top-p 0.95 --min-p 0.05 --metrics`

**Changes from deprecated TurboQuant base:**
- `--cache-type-k q8_0 --cache-type-v q8_0` — TurboQuant not in b9743
- `--flash-attn on` (bare `--flash-attn` silently no-ops in newer builds)
- `--no-cont-batching` — single-user, reduces overhead on 11GB
- `--no-host -fitt 1024` — auto-fit with 1GB margin

## Per-Model Templates

| Model | File | Key additions | n-cpu-moe | Context | Notes |
|-------|------|--------------|-----------|---------|-------|
| Qwen3.6-35B-A3B-MTP | Q4_K_M (22 GB) | `--no-kv-offload --spec-type draft-mtp --spec-draft-n-max 3 -c 230000 -b 2048 -ub 512 --temp 0.2` | **32** | 230K | MTP MoE. KV in sys RAM. |
| Ornith-1.0-35B-Q6-MTP | Q6_K-MTP (28 GB) | `--no-kv-offload --spec-type draft-mtp --spec-draft-n-max 4 -c 131072 -b 2048 -ub 512 --temp 0.6` | **30** | 128K | Frankenstein MTP graft. |
| Qwen-AgentWorld-35B-A3B | UD-Q6_K (28 GB) | `--no-kv-offload -c 131072 -b 2048 -ub 512 --temp 0.6` | **28** | 128K | World model, no MTP. |
| Gemma 4 26B (MoE) | Q4_K_XL (14 GB) | `--no-kv-offload --spec-type draft-mtp --spec-draft-n-max 4 -c 230000 -b 1024 -ub 256 --temp 0.2` | **16** | 230K | Light MoE + MTP. |
| GLM-4.7-Flash | Q4_K_M (18 GB) | `--no-kv-offload -c 200000 -b 1024 -ub 256 --temp 1.0 --repeat-penalty 1.0 --top-k 40` | **24** | 200K | ncm=24 = 36 t/s. |
| Ornith-1.0-9B (dense) | Q6_K (6.9 GB) | `-c 32768 -b 2048 -ub 512 --temp 0.6 --reasoning off` | N/A | 32K | Full GPU. **Raise max_tokens in OWUI** (4096 default truncates long code). |
| Nemotron-Terminal-14B | Q4_K_S (8 GB) | `-c 32768 -b 1024 -ub 256 --temp 0.2` | N/A | 32K | Native 32K. |
| gpt-oss-20b | Q6_K (12 GB) | `--no-kv-offload -c 131072 -b 1024 -ub 256 --temp 0.7` | N/A | 131K | Dense, no MoE. |

**n-cpu-moe on RTX 2080 Ti 11GB (validated June 2026):**
- Q4_K_M MoE (21-23 GB): start **32**, lower to 28 if VRAM allows
- Q6_K MoE (28 GB): start **30**, lower to 28
- Q8_0 MoE (35 GB): min **32** — below OOMs
- 45+ GB MoE: start **42-48**
- Speed delta between values <10% — bottleneck is non-expert layers
- Sweep protocol: `llm-benchmark-optimization` skill → Section 7

## Speculative Decoding Types (b9743)
`--spec-type` accepts: `none, draft-simple, draft-eagle3, draft-mtp, ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod, ngram-cache`

- `draft-mtp` = Multi-Token Prediction (standard, works with MTP heads)
- `draft-eagle3` = EAGLE-3 speculator (requires converted Eagle3 GGUF + matching base model)
- **Important:** The old plain `mtp` type (used by mtp-turbo fork) is NOT supported by upstream builds. Use `draft-mtp` instead.

## Key Pitfalls
- `--spec-type` flag values differ between builds: **b9743 uses `draft-mtp`** (not plain `mtp`). The old mtp-turbo fork used `mtp` — these are NOT interchangeable.
- On b9743, `--spec-draft-model` and `--model-draft` are ALIASES for the same flag (both work).
- Eagle3 speculators are architecture-locked — they will NOT work with any base model except the exact one they were trained for.
- Multi-part GGUFs (Unsloth Dynamic format) must be loaded via the first file only — llama.cpp auto-discovers continuation files.
- `--no-mlock` is NOT a valid flag — it's just the absence of `--mlock`.
- `--chat-template-kwargs` DEPRECATED — use `--reasoning off`

For files >5 GB, the Python `gguf.GGUFReader` often times out. Extract metadata directly from raw bytes:

```bash
timeout 30 python3 -c "
def find_all_metadata(path):
    with open(path, 'rb') as f:
        buf = f.read(2*1024*1024)  # first 2MB
    for needle in [b'architecture', b'block_count', b'head_count', 
                   b'head_count_kv', b'context_length', b'embedding_length',
                   b'rope.freq_base', b'rope.scaling', b'feed_forward_length']:
        pos = 0
        while True:
            found = buf.find(needle, pos)
            if found < 0: break
            chunk = buf[max(0,found-10):min(len(buf),found+50)]
            printable = ''.join(chr(b) if 32 <= b < 127 else f'\\\\x{b:02x}' for b in chunk)
            # Parse adjacent bytes for type+value (GGUF v3 format)
            print(f'{needle.decode()} @ {found}: {printable}')
            pos = found + 1
```"

**GGUF v3 metadata encoding:**
| Offset from key string | Meaning |
|---|---|
| After key + `\x00` | Type: `\x04` = int32, `\x06` = float32, `\x08` = string |
| After type (4 bytes) | Value (4 bytes for int32/float32, uint64 length+data for string) |

To read int32 values: after the type byte (`\x04`), the next 4 bytes are a little-endian int32.

### Forced Context Extension via RoPE Override

For models without YaRN scaling metadata in GGUF (no `rope.scaling.*` fields):
```bash
--override-kv <arch>.context_length=int:<desired_ctx> \
--rope-scaling linear --rope-freq-scale <factor>
```

Where `<factor> = desired_ctx / native_ctx` and `<arch>` from `general.architecture` GGUF field.
- **Format:** `TYPE:VALUE` (e.g., `int:32768`), NOT `KEY=VALUE`
- **Quality caveat:** Linear RoPE scaling on models not trained for it produces token repetition at scale factors >2×. The model will generate but output degrades severely (tested: Moonlight 8K→32K with factor 4.0 repeated "example").

### Dense 24B+ on 11GB VRAM: Speed Expectations

Dense models >15 GB cannot fit fully on 11 GB VRAM. With partial offload:
- 50% layers on GPU → **~5-7 t/s** (PCIe 3.0 x16 bottleneck)
- Compare: MoE models on same GPU → **35-85 t/s** at comparable parameter counts
- Consider: Q4_K_M or Q3_K_L quantization for more speed at lower quality

## KV Cache Budget (11GB)
- MoE 250K: turbo ~9.8GB ✓, q8_0 ~11.5GB ✗
- Dense 9B 128K: ~7.5GB ✓
- Dense 14B 128K: ~12GB ✗ (need `--no-kv-offload`)
- Gemma 26B 128K: 9.0GB ✓
- Gemma 12B 256K: 8.4GB ✓

## Key Pitfalls
- `--chat-template-kwargs` DEPRECATED — use `--reasoning off`
- Gemma 4 does NOT support MTP (crash on startup)
- Always set `--spec-draft-type-k/q` (draft has its own KV cache)
- Turing MTP+Turbo hang: swap to q8_0 cache
- n-cpu-moe = physical cores for CPU-bound MoE (16, not 128)
- `--n-cpu-moe` sets `tensor_buft_overrides` which DISABLES auto-fit AND conflicts with `--cache-type-k/v`. When using --n-cpu-moe you MUST explicitly set -ngl and -c. You CANNOT use --cache-type-k/v or --mlock with --n-cpu-moe on deepseek_v2/v3 architectures (binary ac4cdde). See `references/deepseek-arch-flags.md`.
- For deepseek_v2/v3 architecture models: strip flags to minimum. NO --prio, NO --cont-batching, NO --mlock, NO --no-host -fitt 512, NO batch-size overrides. These all trigger `tensor_buft_overrides` into existing set, blocking auto-fit.
- `hermes config set` can't handle nested list keys

## Reference Files
- `references/flags.md` — Full flag tables, optimization flags, MTP flags, known issues
- `references/build-configs.md` — Build recipe, benchmark results, model downloads
- `references/quantization.md` — KV cache budget tables, cache tradeoffs, quantization notes
- `references/stability-troubleshooting.md` — Models getting stuck
- `references/mtp-research-turing-2025-06-04.md` — Turing MTP hang research
- `references/verified-benchmarks-2025-05-26.md` — Full fleet benchmarks
- `references/hermes-registration.md` — Custom providers setup
- `references/deepseek-arch-flags.md` — DeepSeek v2/v3 / Moonlight architecture flags, tensor_buft_overrides conflict, verified configs
- `references/2026-06-19-fleet-audit-devstral-moonlight.md` — Devstral-24B + Moonlight-16B full GGUF metadata, benchmarks, raw parsing reference, fleet redundancy analysis