# llama-cpp — Quantization Notes

## KV Cache Budget (11GB RTX 2080 Ti)

### MoE Models with `--n-cpu-moe 128`
- **Active params:** ~3GB (constant)
- **KV cache (turbo3/turbo4):** ~0.025MB/token
- **KV cache (q8_0):** ~0.032MB/token
- **OS overhead:** ~0.5GB

| Context | turbo VRAM | q8_0 VRAM | Fits 11GB? |
|---------|-----------|-----------|-----------|
| 128K | 3+3.2+0.5=6.7GB | 3+4.1+0.5=7.6GB | ✓ comfortable |
| 200K | 3+5+0.5=8.5GB | 3+6.4+0.5=9.9GB | ✓ tight |
| 250K | 3+6.3+0.5=9.8GB | 3+8+0.5=11.5GB | turbo ✓, q8_0 ✗ |

### Dense Models
- **9B (Q4_K_M ~5.5GB):** 5.5GB weights + 2GB KV at 128K = ~7.5GB ✓
- **14B (Q4_K_M ~7GB):** 7GB weights + 5GB KV at 128K = ~12GB > 11GB → OOM without `--no-kv-offload`

### Gemma 4 26B (Q4_K_M ~15.6GB)
- Auto-fits ~8.3GB on GPU, ~3.9GB on CPU
- With `--no-kv-offload`: KV cache in system RAM
- At 128K: 9.0GB total VRAM (74%)
- At 200K+: pushes KV past 20GB system RAM

### Gemma 4 12B (Q4_K_M ~6.8GB)
- 256K: 8.4 GiB VRAM (74%). 2.9GB headroom
- 200K: 8.1 GiB
- 131K: 7.3 GiB
- Only +1.1 GiB for doubling context 131K→256K (turbo cache + hybrid attention)

## Cache Type Tradeoffs

| Cache | VRAM at 150K (35B MoE) | Speed vs turbo | Best for |
|-------|------|---------------|----------|
| turbo3/turbo4 | 8.6GB | baseline | **Production** — max context per GB |
| q8_0/q8_0 | +1.4GB (~10GB) | +0% (identical) | Only when headroom >2GB |

**Rule:** start with turbo, switch to q8_0 if VRAM headroom >2GB at target context.

**Turing stability exception:** MTP speculative decoding on RTX 2080 Ti (SM75) with turbo3/turbo4 causes a kernel scheduling hang (llama.cpp #9364, #9866). When this happens, switch to q8_0/q8_0.

## Quantization Notes

### Q4_K_M (Recommended Default)
4.77 BPW. Best balance of speed, quality, and VRAM. Used for all 7 models in fleet.

### Q4_K_XL
Slightly higher quality than K_M (~1-2% perplexity), slightly larger. Used for Gemma 26B before switching to Q4_K_M.

### Q6_K (qwopus-9b)
~1.1 GiB extra VRAM vs Q4_K_M at equivalent context. Costs ~5 t/s generation speed. Used for coding specialist to preserve quality.

### Q1_0 (bonsai-8b)
1-bit extreme quantization (PrismML fork). ~2GB file, ~1GB VRAM, ~150 t/s. For edge/classification tasks only.

### Community Quantized GGUFs
May strip MTP metadata. Weight tensors survive but `nextn_predict_layers` GGUF key may be missing. Check HF `config.json` for `mtp_num_hidden_layers > 0` before assuming MTP-capable.

## The "ngl 99" Trap
`-ngl 99` crams ALL layers into VRAM. For a 22GB file on 11GB, always OOM. Lower `-ngl`:
```bash
# Estimate: VRAM ≈ (file_size / n_layers) * ngl + overhead
# qwen36-35b (22GB, 64 layers): ngl=20 → ~8.9GB (tight but fits)
```