# Fleet Benchmarks — June 20, 2026 (Final, Post-Optimization)

**Binary:** llama.cpp b9743 (c57607016), sm75-optimized (Turing)
**GPU:** RTX 2080 Ti 11GB
**CPU:** Xeon E5-2697A v4, 16C/32T, 64GB RAM
**Cache:** q8_0/q8_0 across all profiles (no TurboQuant in upstream)
**Flags:** `--no-mmap --mlock --prio 2 --flash-attn on --no-host -fitt 512 --metrics`

## Speed & VRAM (Optimized --n-cpu-moe)

| # | Profile | t/s | VRAM | Ctx | Cold Start | Spec | --n-cpu-moe |
|---|---------|-----|------|-----|-----------|------|-------------|
| 1 | qwen36-35b-mtp | **60.0** | 9.2 GB | 250K | 19s | draft-mtp | **32** |
| 2 | gemma-12b | **96.2** | 9.0 GB | 131K | 17s | draft-mtp | n/a dense |
| 3 | gemma4-v2-agentic | **52.7** | 7.7 GB | 131K | 3s | none | n/a dense |
| 4 | gemma-26b-200k | **63.5** | 10.1 GB | 200K | 37s | draft-mtp | 16 |
| 5 | nemotron-term-14b | **56.2** | 10.6 GB | 32K | 5s | none | n/a dense |
| 6 | nemotron-3-nano | **46.8** | ~9 GB | 128K | 35s | none | **38** |
| 7 | qwen3-coder-30b | **38.3** | 10.1 GB | 128K | 37s | none | **42** |
| 8 | qwen3-coder-next-80b | **33.7** | 8.9 GB | 128K | 2:00 | ncm=46 (was 42, OOM'd) | ✅ Stable |
| 9 | glm-4.7-flash-reap | **53.5** | 9.1 GB | 128K | 13s | none | **20** |
| 10 | qwen3-30b-thinking-eagle3 | **35.9** | 10.3 GB | 128K | 45s | draft-eagle3 | 128 |

## --n-cpu-moe Optimization Gains

| Model | Before (ncm=128) | After | Gain | VRAM Used |
|-------|-----------------|-------|------|-----------|
| glm-4.7-flash-reap | 31.1 t/s → **53.5** (ncm=20) | **+72%** 🎯 | 2.6 → 9.1 GB |
| nemotron-3-nano | 36.4 t/s → **46.8** (ncm=38) | +29% | 3.4 → ~9 GB |
| qwen36-35b-mtp | 51.4 t/s → **60.0** (ncm=32) | +17% | 6.8 → 9.2 GB |
| qwen3-coder-30b | 34.6 t/s → **38.3** (ncm=42) | +11% | 7.9 → 10.1 GB |
| qwen3-coder-next-80b | 33.1 t/s → **35.7** (ncm=42) | +8% | 4.4 → 8.4 GB |

**Key finding:** Smaller models (14-23 GB files) benefit most from moving experts to GPU. The GLM gained 72% by putting 28/48 layer experts on GPU. Larger models (45+ GB) gain less because even moving 6/48 layer experts barely changes the CPU/GPU balance.

## Removed Models
- deepseek-v2-lite (redundant: coder-30b covers same role at same speed with 4x context, 3x context)
- qwen3-coder-next-reap (REAP-48B, 32 GB, deleted 2026-06-20)
- qwen3.5-122b-a10b (slow 12.5 t/s, 55 GB, deleted 2026-06-20)

## Perf Comparison: b9743 vs Old b9341 (Pre-Optimization)

| Model | Old t/s | b9743 stock t/s | Optimized t/s |
|-------|---------|-----------------|---------------|
| qwen36-35b-mtp | 45.8 | 51.4 | **60.0** (+31% vs old) |
| gemma-12b | 117.8 | **96.2** | — (known regression #24514) |
| gemma4-v2 (16K) | 54.0 | **52.7** | — |

**Gemma-12b regression:** Known upstream bugs #24514 (25% perf drop b9301→b9305) + #24670 (draft-mtp on Turing sm_75). Not fixable from our side.

## Eagle3 Results

| Model | Without Spec | With Eagle3 | Gain | VRAM |
|-------|-------------|-------------|------|------|
| qwen3-30b-thinking-eagle3 | 34.8 t/s | **35.9 t/s** | +3% | 10.3 GB |
| (n_max=3 p_min=0.75) | | | | |
| qwen3-30b-thinking-eagle3 | 34.8 t/s | **35.5 t/s** | +2% | 10.3 GB |
| (n_max=5 p_min=0.75) | | | | |

Eagle3 provides minimal speedup on CPU-offloaded MoE models because the main model's expert evaluation on CPU is the bottleneck, not token prediction speed. Only useful when the main model is fully GPU-bound.

## Model Roles

- **Fast coding:** qwen3-coder-30b (38.3 t/s, fast cold start)
- **Hard coding:** qwen3-coder-next-80b (35.7 t/s, 2:13 cold start)
- **Primary daily:** qwen36-35b-mtp (60.0 t/s, 250K ctx, MTP)
- **Speed:** gemma-12b (96.2 t/s)
- **Deep reasoning:** gemma-26b-200k (63.5 t/s, 200K ctx)
- **VRAM-light:** glm-4.7-flash-reap (53.5 t/s, 9.1 GB — uses VRAM when active, worth it)
- **Terminal:** nemotron-term-14b (56.2 t/s)
- **Efficiency backup:** nemotron-3-nano (46.8 t/s, ~9 GB)
- **Thinking + Eagle3:** qwen3-30b-thinking-eagle3 (35.9 t/s)