# Local Model Fleet Management — VRAM Budget

## Concrete VRAM Fit Table (11GB GPU)

| Model | File | `--n-cpu-moe` | GPU VRAM | RAM |
|---|---|---|---|---|
| qwen36-35b-mtp | 22 GB | 128 | 7.2 GB | 22 GB |
| nex-n2-mini | 20 GB | 128 | 2.8 GB | 25 GB |
| gemma-26b | 14 GB | 16 | 10.8 GB | 5 GB |

## VRAM Budgeting Formula (MoE models)

For MoE models with `--n-cpu-moe 128`:
- **Active params:** ~3GB (constant, regardless of context)
- **KV cache (turbo):** ~0.025MB per token of context
- **KV cache (q8_0):** ~0.032MB per token of context
- **OS overhead:** ~0.5GB

At 250K context: 3GB (model) + 5GB (turbo cache) + 0.5GB (overhead) = 8.5GB → fits 11GB
At 250K context: 3GB (model) + 6.7GB (q8_0 cache) + 0.5GB = 10.2GB → tight but works

## The "ngl 99" Trap

`-ngl 99` tries to cram ALL layers (including MoE experts) into VRAM. For a 22GB file on an 11GB card, this always OOMs. Lower `-ngl`:

```bash
# Estimate: VRAM_needed ≈ (file_size / n_layers) * ngl_layers + overhead
# For qwen36-35b (22GB file, 64 layers, 11GB VRAM):
#   ngl=30 → (22/64)*30 + ~2GB = ~12.3GB → still too much
#   ngl=20 → (22/64)*20 + ~2GB = ~8.9GB → tight but fits
# Always test empirically.
```

## Cache Type Tradeoffs

| Cache | Speed vs turbo | VRAM cost at 150K (35B MoE) | Best for |
|-------|---------------|------|----------|
| turbo3/turbo4 | baseline | 8.6GB | **Production** — max context per GB |
| q8_0/q8_0 | +0% (identical) | +1.4GB (~10GB) | Only when headroom >2GB |

## Benchmark Methodology

**Never** use inline `python3 -c` with shell variables in f-strings. Write a temp Python file via heredoc:

```bash
cat > /tmp/parse.py << 'PYEOF'
import json
d = json.load(open('/tmp/resp.json'))
print(f"tok/s: {d['usage']['completion_tokens']}")
PYEOF
python3 /tmp/parse.py
```

## Batch Size Optimization

- 35B MoE (with MTP): `--batch-size 2048 --ubatch-size 512`
- 9B dense (with MTP): `--batch-size 1024 --ubatch-size 256`
- Gemma 26B MoE (no MTP): `--batch-size 2048 --ubatch-size 512`

**Important: larger batch is faster on the 2080 Ti for MoE models.** 2048/512 = 48.8 t/s vs 256/256 = 36.6 t/s.

## MTP KV Slot Boundary Bug (GitHub #23658)
MTP draft acceptance rate can collapse at specific context sizes due to KV slot alignment. A 256-token difference (12032 vs 12288) changed acceptance rate from 16% to 71%. Try adjusting context by ±1024 tokens.