# Ray fleet benchmarks on GTX 1080 Ti (May 2026)

Cross-model benchmark notes for Ray's local GGUF fleet on the Dell Precision 5810 / GTX 1080 Ti 11GB using `/usr/local/bin/llama-server` build `3c7616b` (BoFan TurboQuant fork).

## Benchmark method

- Stop the currently active production service first; otherwise free VRAM drops from ~11.0 GiB to ~5.5 GiB and results are invalid.
- Start one fresh server per run.
- Disable prompt cache: `--cache-ram 0`.
- Use one uncached completion request only.
- Tested context: `32768` for cross-model comparison.
- Common tuning unless model-specific flags required:
  - `--cache-type-k turbo4 --cache-type-v turbo4`
  - `--no-kv-offload`
  - `--batch-size 2048 --ubatch-size 512`
  - `--threads 16 --threads-batch 32`
  - `--no-mmap --mlock`

## Verified fleet results

Approximate prompt size varied by tokenizer (~21.7k to 23.1k tokens), so compare prompt speeds directionally and generation speeds more directly.

### Qwen3.6-35B-A3B-UD-Q4_K_S (MoE)
- Prompt: **334.61 tok/s**
- Generation: **29.77 tok/s**
- Prompt tokens: 22,400
- Key quirk: log showed `turbo cache types require flash_attn — enabling automatically` and `llama_context: flash_attn = enabled` even though the command passed `-fa off`.
- Conclusion: best overall generation speed in the fleet; MoE is the strongest match for this hardware.

### Qwen3.5-9B-Uncensored-Q4_K_M
- Prompt: **516.90 tok/s**
- Generation: **13.47 tok/s**
- GPU self: ~5,383 MiB
- Conclusion: prompt-speed king; strongest dense default if you want snappy local behavior.

### Gemma-4-26B-A4B-IQ4_XS (MoE)
- Prompt: **214.39 tok/s**
- Generation: **10.85 tok/s**
- Conclusion: practical big-model alternative; slower than Qwen35A3B but still solid.

### Gemma-3-12B-it-qat-Q4_0
- Prompt: **195.85 tok/s**
- Generation: **9.05 tok/s**
- GPU self: ~7,102 MiB
- Conclusion: slightly better than non-QAT Gemma 3 12B; prefer QAT if keeping one Gemma 3 slot.

### Gemma-3-12B-Q4_K_M
- Prompt: **190.82 tok/s**
- Generation: **8.79 tok/s**
- GPU self: ~7,476 MiB
- Conclusion: works fine, but QAT is the better Gemma 3 variant on this box.

### DeepSeek-R1-Distill-Llama-8B-Q5_K_M
- Prompt: **307.50 tok/s**
- Generation: **7.67 tok/s**
- GPU self: ~5,465 MiB
- Conclusion: useful reasoning specialist, but weaker daily-driver speed than Qwen3.5-9B.

### Ministral-3-8B-Reasoning-2512-Q5_K_M
- Prompt: **282.76 tok/s**
- Generation: **6.67 tok/s**
- GPU self: ~5,774 MiB
- Conclusion: functional but loses clearly to Qwen3.5-9B and DeepSeek-R1-8B on this hardware.

### Qwen3-14B-Q4_K_M
- Prompt: **203.79 tok/s**
- Generation: **5.51 tok/s**
- GPU self: ~8,478 MiB
- Conclusion: fits and runs, but is the weakest dense speed/VRAM trade in the tested fleet.

## Known pitfall from this session

Initial runs were misleading because Qwen3-4B production service was still loaded; that left only ~5.5 GiB free VRAM and caused false OOM/load failures for models that actually fit. Always verify `nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader` before interpreting fitment.

## Session results: May 13, 2026 — 5-model speed test batch

New candidates tested at short context (8192) for generation-only throughput, one at a time, with the same flags: `turbo4`, `ngl=99` (except dense 14B — see below), `batch=2048/512`, `threads=16/32`, `temp=0.7`, `cache_prompt=false`. Short prompt (~44 tokens on Qwen tokenizers, ~16 tokens on Qwen2.5-Coder tokenizer).

### Qwen3-Coder-30B-A3B Q4_K_M (MoE) — 18.6 GB
- Prompt eval: **52.4 tok/s** (44 tok)
- Generation 150: **27.4 tok/s**
- Generation 300: **26.5 tok/s**
- CUDA buffer: ~784 MiB on GPU, 16.7 GiB in pinned host memory
- Conclusion: solid coder addition, matches general 30B MoE speeds

### Qwen3.6-35B-A3B-MTP UD-Q4_K_S (MoE, MTP variant) — 21.4 GB
- Prompt eval: **31.1 tok/s** (44 tok)
- Generation 150: **34.1 tok/s**
- Generation 300: **34.6 tok/s**
- Key observation: **MTP (Multi-Token Prediction) generates faster than prompt eval** — ~34 t/s vs ~31 t/s. This is because MTP predicts multiple tokens per step, so generation throughput actually exceeds single-token prompt processing.
- Conclusion: beats standard 35B generation speed (~30 t/s) by ~15%. If MTP works with Hermes' tool-use pipeline, this is the best generation-speed option.

### Qwen2.5-Coder-14B Q5_K_M (dense) — 10.5 GB
- Prompt eval: **21.6 tok/s** (16 tok — different tokenizer)
- Generation 150: **8.2 tok/s**
- Generation 300: **7.9 tok/s**
- ⚠️ Required `-ngl 25` — Q5_K_M needs 9.5 GB contiguous CUDA buffer. At `-ngl 99`, cudaMalloc fails with OOM on 11 GB card.
- Conclusion: **do not deploy.** Dense 14B at this quant is CPU-bound. A Q4_K_M quant might reach ~15 t/s but still far slower than MoE alternatives.

### Qwen3-30B-A3B-2507 Q4_K_M (MoE) — 18.6 GB
- Prompt eval: **47.6 tok/s** (44 tok)
- Generation 150: **28.3 tok/s**
- Generation 300: **27.7 tok/s**
- Conclusion: essentially identical to Qwen3-Coder-30B. The 2507 date tag doesn't affect throughput — general 30B MoE baseline stands at ~27-28 t/s.

### Qwen3.5-14B-A3B-Claude-Opus-Reasoning-Distilled MXFP4 (MoE) — 8.8 GB
- Prompt eval: **81.3 tok/s** (44 tok)
- Generation 150: **30.7 tok/s**
- Generation 300: **30.5 tok/s**
- Conclusion: **strong addition candidate.** Fastest prompt eval of the batch, matches 30B MoE generation speed, only 8.8 GB footprint. MXFP4 works fine with llama-server MoE flags.

### Tokenizer size difference
The Qwen2.5-Coder-14B tokenizer produced only 16 tokens from the same prompt that Qwen-family tokenizers produced 44 tokens for. This changes prompt-speed comparisons — models should be compared on generation speed (where tokens are counted the same way), not prompt eval speed across different tokenizers.

## Fleet-level takeaways (updated May 13)

- On this machine, **MoE models are the sweet spot**.
- Best overall local runner: **Qwen3.6-35B-A3B**.
- Best dense default: **Qwen3.5-9B**.
- Best Gemma 3 slot: **Gemma-3-12B-QAT**.
- Qwen3-14B is viable but not compelling operationally versus 9B or MoE options.
- TurboQuant KV + current fork can silently/implicitly force Flash Attention; always confirm behavior from logs.