# VRAM Breakdown: Dense 14B on GTX 1080 Ti (11 GB)

## Problem

`qwen2.5-coder-14b-instruct-q4_k_m.gguf` (dense 14B, Q4_K_M) fails to load at 128K context and even at 65K context on a **GTX 1080 Ti (11163 MiB VRAM)** with `-ngl 99`.

## Root Cause

Newer llama.cpp builds (b1-3c7616b) include a strict **VRAM fitting check** (`common_params_fit_impl`) that aborts if the projected total exceeds free VRAM. The old build did not enforce this as strictly, which is why coder14 worked before.

### Memory Breakdown at 65K (with Q4 KV cache)

```
| memory breakdown [MiB]  | total    free     self   model   context   compute    unaccounted |
|   - CUDA0 (GTX 1080 Ti) | 11163 = 11023 + (11571 =  8148 +    3264 +     158) +      -11430 |
```

| Component | Size (MiB) | Notes |
|---|---|---|
| Model weights (Q4_K_M) | **8,148** | 48 layers, 5120 embedding dim |
| KV cache (65K, Q4) | **3,264** | `--cache-type-k turbo4 --cache-type-v turbo4` = 0.5 bytes/elem |
| Compute buffers | **158** | Batch-size dependent |
| **Total required** | **11,571** | |
| **Free VRAM** | **11,023** | |
| **Shortfall** | **389** | llama.cpp wants 1024 MiB buffer, but even raw fit fails |

At **128K**: KV cache alone would be ~6,528 MiB (double), pushing total to ~14,834 MiB — way over 11 GB.

## The Fix: `--no-kv-offload`

Adding `--no-kv-offload` keeps the KV cache in **system RAM** instead of VRAM. This frees the 3,264 MiB of KV cache, leaving only model weights (8,148 MiB) + compute buffers (158 MiB) = **~8,306 MiB** — well within 11,023 MiB free.

```yaml
# Working config for dense 14B on 11GB VRAM
cmd: /usr/local/bin/llama-server \
  -m /models/llm/coder14/qwen2.5-coder-14b-instruct-q4_k_m.gguf \
  -ngl 99 \
  --no-kv-offload \
  --cache-type-k turbo4 --cache-type-v turbo4 \
  -c 65536 --ctx-size 65536 \
  --batch-size 1024 --ubatch-size 256
```

## Performance with `--no-kv-offload` at 65K

| Metric | Value |
|---|---|
| Prompt eval | **242.3 t/s** |
| Generation | **22.3 tok/s** |
| Model | qwen2.5-coder-14b Q4_K_M (dense) |

Prompt eval speed is unaffected since the KV cache is written during eval regardless of location. Generation speed takes a small hit (CPU-side KV cache reads during autoregressive decode add latency) but remains usable at 22 tok/s.

## Comparison: MoE vs Dense

| Architecture | Model Weights | KV Cache (128K, Q4) | Total | Fits 11 GB? |
|---|---|---|---|---|
| MoE 35B (Qwen3.6) | ~1-2 GB (shared layers only) | ~3-4 GB | ~5-6 GB | ✅ Yes (experts in RAM) |
| Dense 14B (Qwen2.5) | 8.1 GB | 6.4 GB | 14.5 GB | ❌ No (even at 65K fails without no-kv-offload) |

MoE models are inherently more VRAM-efficient for high-context workloads because:
- Only the shared layers and router are offloaded to GPU
- Expert weights stay in system RAM, loaded on demand via `--n-cpu-moe N`

Dense models must load ALL parameters to VRAM with `-ngl 99`, leaving little room for KV cache.