# Ray: Dense vs MoE download heuristics on GTX 1080 Ti (11GB)

Use this when the user wants to download larger GGUFs for Ray's local fleet and asks whether they will actually be fast on the 1080 Ti.

## Key practical conclusion

On this box, **MoE can beat smaller dense models by a huge margin**.

Reason:
- `Qwen3.6-35B-A3B` is a 35B MoE with only ~3B active parameters per token.
- Dense 14B / 24B / 32B models must push all layers every token.
- On GTX 1080 Ti + llama.cpp, memory bandwidth and partial CPU spill make dense models much slower than their parameter counts suggest.

So it is **expected**, not suspicious, for `Qwen3.6-35B-A3B` to outrun smaller dense models.

## Benchmarked anchor points on Ray's box

Already-measured local results:
- `Qwen3.6-35B-A3B-UD-Q4_K_S`: ~`29.8 tok/s` generation
- `Qwen3.5-9B-UD-Q4_K_M`: ~`13.5 tok/s` generation
- `Qwen3-14B-Q4_K_M`: ~`5.5 tok/s` generation
- `DeepSeek-R1-8B-Q5_K_M`: ~`7.7 tok/s` generation
- `Qwen2.5-Coder-14B Q5_K_M`: ~`8.2 tok/s` generation **(ngl=25 only — see below)**

## Dense 14B Q5_K_M specific VRAM limitation (May 13 test)

The Qwen2.5-Coder-14B Q5_K_M (10.5 GB file) requires a **9.5 GB contiguous CUDA buffer** for full GPU offload. On Ray's GTX 1080 Ti (11 GB):

- With the daily production server (Qwen3.6-35B) running: ~6.7 GB used, ~4.5 GB free → **fails with cudaMalloc OOM**
- With daily stopped: ~250 MB used, ~10.9 GB free but `load_tensors` still fails because 9.5 GB > what's available after the loading process itself needs headroom
- After all: `-ngl 99` for this model is impossible on this hardware

Workaround: `-ngl 25` loads 25/49 layers on GPU (5.0 GB CUDA buffer) and leaves 24 layers on CPU. Result: generation drops to **~8 tok/s** — CPU-bound.

## Dense 14B Q4_K_M confirmed fit (also May 13)

**The Q4_K_M quant of Qwen2.5-Coder-14B (8.4 GB file, Qwen/Qwen2.5-Coder-14B-Instruct-GGUF) does fit with `-ngl 99`.** Verified:

- CUDA0 model buffer: **8,148 MiB** (8.1 GB)
- Context + compute overhead: ~725 MiB
- Total CUDA usage: ~8,873 MiB (out of 11,163 MiB on GTX 1080 Ti)
- Free VRAM after load: **1,883 MiB** (tight but functional for production use)
- Host pinned memory: only 417 MiB (most weights live on GPU)

**Speed at ngl=99: ~29.2 tok/s generation** — comparable to the 30B MoE models and dramatically better than the Q5_K_M at ngl=25 (8 tok/s).

**Recommendation:** Prefer Q4_K_M for dense 14B models on this hardware. The Q5_K_M is not usable for interactive speeds. MoE 30B models remain the stronger choice overall (~27-35 t/s with only ~784 MB CUDA buffer), but if a dense 14B is needed, Q4_K_M at ngl=99 is viable.

## Speed expectations before benchmarking
- Dense 14B is already much slower than the MoE 35B on this hardware.
- Dense 24B/32B should be treated as **specialist slow models**, not default interactive drivers.

## Download strategy for new large dense models

Prefer **fit-biased quants**, not ideal-on-paper quants.

Downloaded choices that match the hardware:
- `Mistral-Small-3.1-24B-Instruct-2503.Q3_K_M.gguf`
- `Mistral-Small-3.1-24B-Instruct-2503.mmproj-Q8_0.gguf`
- `qwen2.5-coder-32b-instruct-q2_k.gguf`
- `DeepSeek-R1-Distill-Qwen-32B-Q2_K.gguf`

Why these quants:
- `24B dense`: `Q3_K_M` is the realistic starting point on 11GB VRAM.
- `32B dense`: `Q2_K` is the realistic starting point if the user still wants to test it.
- `Q4_K_M`-class quants for 24B/32B are generally too heavy for a practical 11GB experience once runtime overhead is included.

## Speed expectations before benchmarking

Use these as planning estimates, not guarantees:
- `Mistral-Small-3.1-24B Q3_K_M`: roughly `3–5 tok/s`
- `Qwen2.5-Coder-32B Q2_K`: roughly `2–4 tok/s`
- `DeepSeek-R1-Distill-Qwen-32B Q2_K`: roughly `2–3.5 tok/s`

These are **not speed upgrades** over Ray's current winners.

## Recommendation framing for Ray

Be direct:
- These bigger dense downloads are **quality experiments / specialist models**.
- They are **not expected to beat** `Qwen3.6-35B-A3B` for speed.
- They are also unlikely to beat `Qwen3.5-9B-UD` for dense-model responsiveness.

Suggested framing:
- `Mistral Small 24B` → likely the most interesting of the slow dense additions
- `Qwen2.5-Coder-32B` → specialty code model, probably slow
- `DeepSeek-R1-Distill-Qwen-32B` → specialty reasoning model, probably slow + verbose

## Operational rule

When a user asks for "download the bigger ones too," do **not** default to the prettiest quant label.
Choose the **lowest quant that still makes the experiment meaningful** on the actual hardware, then benchmark before recommending service integration.