# Dense Model Max-Context Calculation

Unlike MoE models, **dense models cannot use `--n-cpu-moe`** — all weights go to GPU per `-ngl`. This makes context budget calculation simpler but tighter.

## The Formula

```
max_tokens = (total_VRAM_MiB - file_size_MiB - overhead_MiB) / bytes_per_token
```

Where:
- `total_VRAM` = 11264 MiB for RTX 2080 Ti
- `file_size` = the GGUF file size (stat --format="%s" /path/model.gguf / 1048576)
- `overhead` = 400-500 MiB (CUDA context, scratch buffers, -fitt margin)
- `bytes_per_token` = 2 × n_layers × n_kv_heads × head_dim × bytes_per_elem

Cache bytes per element: q8_0 = 1, q4_0 = 0.5

## The Real Bottleneck: Compute Graph Buffers

**The formula above underestimates VRAM needs.** The actual bottleneck is NOT the KV cache allocation — it's the **compute graph buffers** (~350-400 MiB) that llama.cpp needs for intermediate tensors during decode.

A model can have enough VRAM for weights + KV cache but still crash with:
```
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 396.08 MiB on device 0: cudaMalloc failed: out of memory
```

These buffers scale with `--batch-size` and `--ubatch-size`. **Shrinking batch/ubatch** can free compute buffer space and unlock +1-2K of context:

| Batch/Ubatch | Compute Buffer Size | Context Headroom |
|-------------|-------------------|-----------------|
| 1024/256 | ~400 MiB | Low |
| 512/128 | ~370 MiB | Medium |
| 256/64 | ~350 MiB | High |
| 128/32 | ~340 MiB | Highest |

**Additionally, `--no-warmup` skips the warm-up decode that needs these buffers** — essential when total VRAM is borderline.

**Rule of thumb:** After weights + KV cache allocation, reserve at least **350 MiB** for compute graph buffers on 11GB. If the model + KV fits but `cudaMalloc` fails during `graph_reserve` or `process_ubatch`, reduce batch/ubatch or add `--no-warmup`.

## Example: Mistral Nemo 12B (Q4_K_M, 7.0 GB)

- file_size: 7,477 MB → 7,477 MiB
- layers: 40, KV heads: 8, head_dim: 128
- q4_0 KV: 40 × 2 × 8 × 128 × 0.5625 (q4_0 block overhead) = 46,080 bytes/token
- Available: (11264 - 7477 - 500) = 3,287 MiB (raw formula)
- Max tokens (raw): 3,287 MB / 45 KB = ~73K (formula under-estimates — see above)

**Actual tested results:**
| Context | Batch/Ubatch | VRAM | Result | Gen Speed |
|---------|-------------|------|--------|-----------|
| 80K (81920) | 1024/256 | 10,920 MiB | ✓ Works | 107 t/s |
| 81K (82944) | 1024/256 | 10,970 MiB | ✓ Works | — |
| **82K (83968)** | **256/64** | **10,974 MiB** | **✓ Max** | **62.5 t/s** |
| 82K (83968) | 1024/256 | — | OOM (graph buffer) | — |
| 83K (84992) | 128/32 | — | OOM (graph buffer) | — |
| 84K+ | any | — | OOM | — |

**`--no-warmup` required** at 81K+ to avoid OOM during first decode.

## Example: Phi-4 15B (Q3_K_M, 6.9 GB)

- file_size: 7,022 MB → 7,022 MiB
- layers: 48 KV heads: 8, head_dim: 128
- q4_0 KV: 48 × 2 × 8 × 128 × 0.5625 (q4_0 block overhead) = 55,296 bytes/token
- Available: (11264 - 7022 - 500) = 3,742 MiB (raw formula)

**Actual tested results:**
| Context | Batch/Ubatch | VRAM | Result | Gen Speed |
|---------|-------------|------|--------|-----------|
| **64K (65536)** | **512/128** | **10,938 MiB** | **✓ Stable** | **47-50 t/s** |
| **65K (66560)** | **128/32** | **10,978 MiB** | **✓ Max** | **46.8 t/s** |
| 65K (66560) | 256/64 | — | OOM (graph buffer) | — |
| 66K+ | any | — | OOM | — |

**`--no-warmup` required** at 65K.

## Key Differences from MoE

| Aspect | Dense | MoE |
|--------|-------|-----|
| GPU weights | ~100% of GGUF file | ~30-35% (non-expert only) |
| Context-limited by | File size (big) | KV cache (bigger) |
| Mitigation | Lower quant or lower ctx | --n-cpu-moe + lower ctx |
| Speed on 11GB | 47-107 t/s | 33-165 t/s |

## Quick Check Command

```bash
python3 -c "
# Adjust for your model
file_mb = 7477  # from 'stat --format=\"%s\" model.gguf'
layers = 40
kv_heads = 8
head_dim = 128
cache_bytes = 0.5625  # q4_0 with block overhead = 0.5625, q8_0 = 1

bytes_per_token = 2 * layers * kv_heads * head_dim * cache_bytes
avail = 11264 - file_mb - 500
# Subtract ~350 MiB for compute graph buffers
avail -= 350
max_tok = int(avail * 1024 * 1024 / bytes_per_token)
print(f'KV: {bytes_per_token/1024:.0f} KB/token')
print(f'Available: {avail} MiB')
print(f'Max context: {max_tok} tokens ({max_tok//1024}K)')
print(f'NOTE: Actual max may be +/-2K depending on batch-size and --no-warmup')
"
```