# llama-cpp — Flag Reference for RTX 2080 Ti (SM75, 11GB)

## Universal Base Flags
Every model starts with these:
```
--cache-type-k turbo3 --cache-type-v turbo4 -ngl 99 --no-mmap --mlock --jinja
--reasoning off --prio 2 --poll 80 --cont-batching --timeout 300
--threads 16 --threads-batch 32 --parallel 1
--temp 0.2 --top-p 0.95 --min-p 0.05 --top-k 20 --metrics
```

**Pitfall:** `--chat-template-kwargs '{"enable_thinking":false}'` is DEPRECATED. Use `--reasoning off`. Confirmed on build b9341.

## MoE Models (Qwen 35B/30B, with MTP)
```
-c 250000 --ctx-size 250000
--batch-size 2048 --ubatch-size 512
--n-cpu-moe 128
--flash-attn on
--spec-type mtp --spec-draft-n-max 4
```
- threads=16 faster than 32 for 35B MoE (48.8 vs 37.0 t/s)
- batch 2048/512 optimal (48.8 t/s vs 36.6 at 256/256)
- draft=4 optimal for all contexts (48.8 vs 38.7 @3, 37.8 @2)

## Non-MoE Models (9B, with MTP)
```
-c 200000 --ctx-size 200000
--batch-size 1024 --ubatch-size 256
--no-kv-offload
--flash-attn on
--spec-type mtp --spec-draft-n-max 4
```

## Gemma 4 26B A4B MoE (CPU-bound)
```
-c 131072 --ctx-size 131072
--batch-size 1024 --ubatch-size 256
--n-cpu-moe 16
--no-kv-offload
--cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on
--no-cont-batching
--no-host -fitt 512
--threads 16 --threads-batch 16
```

Key differences from Qwen MoE:
| Flag | Gemma 26B | Why |
|------|-----------|-----|
| n-cpu-moe | **16** | Physical cores. NOT 128/256 (oversubscription slows). n-cpu-moe=8 also crashes (OOM) |
| threads-batch | **16** | Match physical cores, not 32 |
| batch-size | **1024** | Reduce VRAM pressure (8.3GB on GPU) |
| cache-type | **q8_0** | Universal safety on Turing. Turbo might hang |
| cont-batching | **--no-cont-batching** | Single-user, no overhead needed |

## Gemma 4 12B Dense
```
-c 256000 --ctx-size 256000
--batch-size 1024 --ubatch-size 256
--cache-type-k turbo3 --cache-type-v turbo4
--flash-attn on --no-host -fitt 512
--threads 16 --threads-batch 32
```
- 53.7 t/s, 8.4 GiB VRAM at 256K
- top_k=64 (Google recommends)
- No MTP support (Gemma4 architecture has no MTP head)
- `--reasoning off` REQUIRED for agent workflows

## DeepSeek v2/v3 / Moonlight (MLA MoE)

**WARNING:** These architectures have `tensor_buft_overrides` conflicts. See `references/deepseek-arch-flags.md` for full details and multiple config options.

**Fast config (with --n-cpu-moe, stripped flags):**
```
--jinja --reasoning off -ngl 99 --no-mmap
-c 32768
--n-cpu-moe 16
--cache-type-k q8_0 --cache-type-v q8_0
--flash-attn on
--temp 0.2 --top-p 0.95 --min-p 0.05
--metrics
```
- 33.9 t/s at 10.2 GB (tight)

**Stable config (no --n-cpu-moe, auto-fit):**
```
--jinja --reasoning off --no-mmap --mlock
--prio 2 --poll 30 --timeout 300
--threads 16 --threads-batch 32 --parallel 1
-c 65536 --batch-size 1024 --ubatch-size 256
--temp 0.2 --top-p 0.95 --min-p 0.05
--metrics
```
- 24.9 t/s at 9.5 GB (stable)

**DO NOT USE with DeepSeek arch on current build (ac4cdde):**
- `--mlock` with `--n-cpu-moe` — triggers override conflict
- `--prio` with `--n-cpu-moe` — triggers override conflict
- `--no-host -fitt 512` with `--n-cpu-moe` — triggers override conflict
- `--cont-batching` with `--n-cpu-moe` — triggers override conflict

## Optimization Flags

### TurboQuant KV Cache Tuning
| Type | Bits | Use case |
|------|------|----------|
| turbo4 | 4-bit | Default for V cache (attention values) |
| turbo3 | 3-bit | Default for K cache (attention keys) |
| turbo2 | 2-bit | Extreme VRAM pressure only |

Recommended: `--cache-type-k turbo3 --cache-type-v turbo4`

### Flash Attention
`--flash-attn on` for sm_75 (Turing tensor cores handle FA efficiently).

### CUDA Graphs
Built with `GGML_CUDA_GRAPHS=ON`. No CLI flag needed.

### Memory/VRAM Optimization (11GB)
- `--no-host` — bypass host buffer, frees extra VRAM
- `-fitt 512` — reduce safety margin from 1GB to 512MB
- `--op-offload` — keep on (offload host tensor ops to device)
- `--no-repack` — disable weight repacking. Saves 5-10% VRAM at 5-10% speed cost. Use only if OOM.

### Prompt Caching
- `--cache-prompt` — enable prompt caching (default: on)
- `--cache-reuse 256` — reuse prompt KV cache chunks
- `--cache-ram 16384` — increase cache RAM (if spare system RAM)

### Server Performance
- `--threads-http 4` — limit HTTP threads, reduce CPU contention
- `--parallel 1` — single-user max throughput

### MTP Speculative Decoding Flags
```
--spec-type mtp
--spec-draft-n-max 4              # 4 is optimal on 2080 Ti
--spec-draft-n-min 1
--spec-draft-p-min 0.75
--spec-draft-p-split 0.10
--spec-draft-type-k q4_0          # CRITICAL: draft K cache type
--spec-draft-type-v q4_0          # CRITICAL: draft V cache type
```

## Known Issues & Workarounds
1. MTP + `-sm tensor`: crashes
2. MTP + low temp (<0.1): acceptance rate drops. Use ≥0.2
3. `--chat-template-kwargs` is DEPRECATED — use `--reasoning off`
4. Gemma 4 does NOT support MTP — crash on startup with "MTP not supported for trunk architecture 'gemma4'"
5. MTP draft model has its OWN KV cache — always set `--spec-draft-type-k/q` to q4_0 or turbo
6. n-cpu-moe oversubscription on CPU-bound MoE: n-cpu-moe should match **physical CPU cores**, not expert count
7. Turing (SM75) MTP+Turbo cache silent hang: swap to `--cache-type-k q8_0 --cache-type-v q8_0`
8. `hermes config set` can't handle nested list keys — use Python yaml.dump
9. **DeepSeek v2/v3 `tensor_buft_overrides` conflict:** `--n-cpu-moe` sets overrides that block auto-fit and conflict with `--cache-type-k/v`, `--mlock`, `--prio`, `--cont-batching`. Solution: either strip flags (fast, fragile) or drop `--n-cpu-moe` (slower, stable). See `references/deepseek-arch-flags.md`.
10. **Model architecture research:** When uncertain about a flag's behavior, look up the source code on GitHub (`common/arg.cpp`, `common/common.h`) — don't guess or estimate. The `--n-cpu-moe` regex target is `\.ffn_(up|down|gate|gate_up)_(ch|)exps` in `common/common.h`.