---
name: vram-estimation
description: Calculate VRAM budget for any GGUF model on a given GPU — model weights + KV cache + overhead. Critical distinction between MoE and dense models.
---

# VRAM Estimation for llama.cpp

## The Formula

```
Total VRAM = Model_Weights_On_GPU + KV_Cache + Overhead
```

Where each component depends critically on model type (MoE vs dense) and flags.

## Model Weights: Dense vs MoE

### Dense Models
All weights go to GPU per `-ngl`. VRAM = GGUF file size (approximately).

### MoE Models (`--n-cpu-moe` CRITICAL)
**`--n-cpu-moe` keeps expert weights in system RAM, NOT GPU VRAM.**
Only non-expert layers go to GPU.

**Rule of thumb:** Non-expert layers = ~30-35% of total weights for large MoE, ~40% for small MoE.
- Attention projections (Q, K, V, O)
- Embedding + output layers
- Layer norms, router
- Shared experts

**DANGER:** Computing VRAM from GGUF file size alone for MoE models gives wildly wrong results.
- Gemma 26B Q4_K_XL: 15.6 GB file → only ~9.6 GB VRAM (with `--n-cpu-moe 16`)
- DeepSeek-Coder-V2-Lite Q5_K_M: 11.85 GB file → only ~4-5 GB VRAM (with `--n-cpu-moe 16`)
- Nemotron-3-Nano-30B Q4_K_M: 9.35 GB file → only 3.4 GB VRAM (with `--n-cpu-moe 128`)
- **Qwen3-Coder-Next Q4_K_M: 46 GB file → fits at `--n-cpu-moe 46` + 140K ctx (33.7 t/s, validated); OOMs at `--n-cpu-moe 42` + 128K ctx** (11GB card)
- **Qwen3.6 FableVibes 14B-A3B Q4_K_M: 7.9 GB file → only ~1 GB on GPU. KV cache dominates. 256K at q4_0 fits (8.2 GB total, ~91 t/s). 256K at q8_0 OOMs (KV alone ~7.2 GB). 192K at q8_0 works (~6.4 GB total, 130 t/s)** (11GB card)

## --n-cpu-moe Tuning: VRAM vs Speed Tradeoff

Each point of --n-cpu-moe moved from CPU to GPU adds approximately one layer's worth of expert weights.
For a model at Q4_K_M, each layer's experts cost roughly:
- 128-expert MoE (35B): ~200 MB/layer
- 512-expert MoE (80B): ~830 MB/layer  
- Small MoE (23B GLM): ~240 MB/layer

**Observed VRAM increments from lowering --n-cpu-moe on 11GB card:**

| Model | ncm=128 VRAM | Optimized ncm | Optimized VRAM |
|-------|-------------|---------------|----------------|
| Qwen3-Coder-Next Q4_K_M (46 GB, 48 layers, 512 experts) | N/A | **48** (all MoE CPU) | ~9.6-10.6 GB ✓ at 32K ctx |
| Qwen3-Coder-Next Q4_K_M (same) | N/A | **46** (2 layers' MoE on GPU) | ~8.9 GB ✓ at 128K ctx (validated: 33.7 t/s) |
| Qwen3-Coder-Next Q4_K_M (same) | N/A | 42 (6 layers' MoE on GPU) | ~14.7 GB **OOM** at 128K ctx |

**Key rule for 11GB card with 46 GB+ MoE files:** Set `--n-cpu-moe` to at least `(layer_count - 2)` to leave headroom for 128K+ KV cache. Each layer's MoE on GPU adds ~830 MB. At `--n-cpu-moe 46` (2 layers on GPU), total VRAM with 128K q8_0 KV cache is ~8.9 GB. At `--n-cpu-moe 42` (6 layers on GPU at ~13+ GB), VRAM exceeds 11 GB and OOMs. When in doubt, measure with the actual context length you use — KV cache often dominates. See `local-model-fleet-management` for benchmark results at each --n-cpu-moe value.

### Standard MHA/GQA
Per layer per token: `2 * n_kv_heads * (hidden / n_heads)`
- 8 KV head × 128 head_dim: 2048 values/layer
- 16 KV head × 128 head_dim: 4096 values/layer
- Gemma 12B (dense, 48 layers): ~98,304 values/token → 98 KB/token at q8_0
- At 256K: ~25 GB (q8_0) → use turbo3/turbo4 (~12.5 GB)

### MLA (Multi-head Latent Attention — DeepSeek)
Per layer per token: `kv_lora_rank + qk_rope_head_dim`
- DeepSeek-Coder-V2-Lite: 512 + 64 = 576 values/layer
- 27 layers: 15,552 values/token → 15.5 KB/token at q8_0
- At 128K: ~2 GB (q8_0) vs ~12.5 GB for standard

### 2 KV Head models (GQA 8:1)
Per layer per token: `2 * 2 * head_dim`

**Nemotron-3-Nano** (52 layers, unknown head_dim, ~small)
- Measured: 3.4 GB total VRAM at 128K (including model weights)

**Qwen3-Coder-Next** (48 layers, 2 KV heads, **256 head_dim**)
- Per token: 2 × 2 × 256 × 48 = **49,152 bytes (48 KB/token) at q8_0**
- At 32K context: 1.57 GB · At 64K: 3.15 GB · At 128K: 6.29 GB

**⚠️ Hybrid architectures (DeltaNet + Attention)** — Qwen3-Coder-Next's 48 layers are arranged as 12×(3×DeltaNet→MoE + 1×Attention→MoE). Only 12 attention layers produce KV, the 36 DeltaNet layers don't. However, llama.cpp allocates KV cache buffer for ALL layers regardless of architecture, treating non-attention layers as zero-filled. **The VRAM allocation equals the all-layers worst case (48 KB/token)** even though runtime cache pressure is lower. Always budget from the full layer count for allocation, use actual attention-layer count for runtime calculations.

**Qwen3.6 FableVibes** (REAP-pruned 14B-A3B, qwen35moe arch, ~28 layers, ~4 KV heads, 128 head_dim)
- Per token: 2 × 4 × 128 × 28 = **28,672 bytes (28 KB/token) at q8_0; 14 KB/token at q4_0**
- At 128K q8_0: 3.6 GB · At 192K q8_0: 5.4 GB · At 256K q8_0: 7.2 GB (OOMs on 11GB) · At 256K q4_0: 3.6 GB ✓
- Non-expert weights on GPU: ~1 GB (tiny — most params are pruned experts on CPU via `-ngl 99` with no `--n-cpu-moe` needed since experts are already in CPU by default)

## Cache Type Size Factor
| Type | Bits | Factor vs f16 |
|------|------|--------------|
| q8_0 | 8-bit | 0.5× f16 |
| q4_0 | 4-bit | 0.25× f16 |
| turbo4 | 4-bit | 0.25× f16 |
| turbo3 | 3-bit | 0.1875× f16 |
| turbo2 | 2-bit | 0.125× f16 |

## Overhead
- OS/driver: ~300-500 MB
- llama.cpp scratch buffers: ~200-500 MB
- `-fitt 512` reduces safety margin to 512 MB
- Use `--no-host` to free host buffer (~200 MB)

## Quick Check: Does It Fit?

1. Check if MoE → GPU weight ≈ 0.3 × GGUF file size; else GPU weight ≈ file size
2. Calculate KV cache: tokens × values_per_token × cache_factor
3. Add overhead: 0.5 GB
4. Compare to `nvidia-smi` reported memory (11,264 MiB for 11GB card)

## Reality Check: Dense Models >15 GB on 11 GB VRAM

A dense 24B model at Q5_K_XL (15.6 GB) cannot fully fit in 11 GB VRAM:
- Max GPU offload: ~20-22 layers of 40 (~50% of weights)
- Remaining layers on CPU → PCIe 3.0 x16 bottleneck → **5-7 t/s max**
- Compare: MoE models on same GPU → 35-85 t/s
- Mitigation: lower quantization (Q4_K_M saves ~3 GB), or accept the speed limit

## Forced Context Extension: VRAM Impact

When using `--override-kv <arch>.context_length=int:<N>` with `--rope-scaling linear`:
- KV cache scales linearly with N (standard GQA) or negligibly (MLA)
- Compute buffers scale up significantly — observed +4 GB on Moonlight from 8K→32K despite MLA having negligible KV cache
- The extra VRAM comes from expanded internal buffers, NOT KV cache
- Budget: +3-5 GB VRAM for 4× context extension on non-MLA architectures

## Compute Graph Buffer Bottleneck

At max context on limited VRAM (11GB), the **compute graph buffers** (~340-400 MiB) fail before the KV cache does:

```
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 396.08 MiB: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 415323264
```

These buffers are for intermediate tensors during decode and scale with `--batch-size`/`--ubatch-size`. Strategy when hitting this error:

1. **Reduce batch/ubatch:** halving batch roughly halves the compute buffer. Try 256/64 or 128/32.
2. **Add `--no-warmup`:** skips the first decode that triggers the allocation.
3. **Trade speed for context:** Smaller batch reduces gen speed (~10-30%) but can unlock +1-2K context.

**Example result:** Mistral Nemo 12B went from 80K→82K (+2K), Phi-4 15B from 64K→65K (+1K) on 11GB by using `--batch-size 256 --ubatch-size 64 --no-warmup` instead of the default 1024/256.

See `references/dense-model-max-context.md` for full methodology and benchmark tables.

## References
- `references/qwen3-coder-next-specs.md` — Full dimensions, VRAM budget tables, per-context KV cache breakdown for Qwen3-Coder-Next hybrid MoE
- `references/dense-model-max-context.md` — Max-context calculation for dense models on 11GB (Mistral Nemo 12B, Phi-4 15B methodology)
- `inference/llama-cpp/references/quantization.md` — Cache budgets per model
- `inference/llama-cpp/references/flags.md` — Optimization flags for VRAM