# Fable5 MoE Prefetch Patches (thecodacus/llama.cpp)

**Source:** [thecodacus/llama.cpp](https://github.com/thecodacus/llama.cpp), branch `fable5/prefetch-experts`
**YouTube:** [I Asked Claude Fable 5 to Improve llama.cpp.. and It Did](https://youtu.be/VytSYCDhWQ0)
**Applied to our build:** 2026-07-05, commits `4276dac` and `c544a27` (cherry-picked onto b9743 base `c576070`)

## What They Do

Two patches that overlap MoE expert weight uploads (CPU→GPU over PCIe) with GPU compute during prefill. At large batch sizes where nearly all experts are selected, the router-ids readback is skipped and full expert tensors are uploaded through a second CUDA stream while the previous layer's compute runs.

### Commit 1: Overlap (1163cb349 → 4276dac)
- Adds `GGML_SCHED_PREFETCH_EXPERTS` env var (opt-in, default off)
- Creates a second CUDA backend instance for async uploads
- 2 staging slots with event-ordered synchronization
- Modifies only `ggml/src/ggml-backend.cpp` (+102 lines)

### Commit 2: 3-slot fix + UAF fix (5f83fbbe7 → c544a27)
- Expands from 2 to 3 prefetch slots (covers gate/up/down per MoE layer)
- Fixes use-after-free: on OOM during slot regrow, staging tensor pointers that were repointed kept dangling into freed device memory
- Slots sized once from graph max; allocation happens before freeing
- Env var value controls slot count; default 3, max 8
- Modifies only `ggml/src/ggml-backend.cpp` (+90/-19 lines)

## How to Apply to Future Builds

```bash
cd ~/src/llama.cpp

# Add fork as remote if not present
git remote add codacus https://github.com/thecodacus/llama.cpp.git
git fetch codacus fable5/prefetch-experts

# MUST be on a base that supports our models (c576070 = b9743)
# Wrong base (e.g. 048a490) = SSM tensor missing → model load fails
git reset --hard c576070
git cherry-pick 1163cb349   # overlap
git cherry-pick 5f83fbbe7   # 3-slot + UAF fix

# Full clean rebuild (rm -rf build is mandatory after reset)
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=75 \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-13 \
  -DCMAKE_C_COMPILER=/usr/bin/gcc-13 -DCMAKE_CXX_COMPILER=/usr/bin/g++-13 \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc) --config Release --target llama-server
sudo cp build/bin/llama-server /usr/local/bin/llama-server-sm75
sudo cp build/bin/*.so* /usr/local/lib/ && sudo ldconfig
```

## Benchmark Results (RTX 2080 Ti, sm_75, PCIe Gen3)

**Setup:** Qwen3.6-35B-A3B-MTP Q4_K_M, `--no-mmap --mlock`, `--n-cpu-moe 32`

| Config | ubatch | Tokens | t/s | vs Baseline |
|--------|--------|--------|-----|-------------|
| Baseline | 512 | 1808 | **293** | — |
| Prefetch | 512 | 1806 | **219** | -25% |
| Baseline | 2048 | 3251 | **373** | — |
| Prefetch | 2048 | 3303 | **360** | -3% |

### Proof the prefetch IS activating

A temporary diagnostic log was added at the threshold condition to verify:
```
[PREFETCH-DIAG] HIT slot=0 ids_ne0=8 ids_ne1=1838 n_expert=256 threshold=512 size=184549376
[PREFETCH-DIAG] HIT slot=1 ids_ne0=8 ids_ne1=1838 n_expert=256 threshold=512 size=184549376
[PREFETCH-DIAG] HIT slot=2 ids_ne0=8 ids_ne1=1838 n_expert=256 threshold=512 size=184549376
```

All 3 slots fire correctly. 176 MiB per expert tensor, all conditions met. The code works — it just doesn't help.

### Why it doesn't help: compute:upload inversion

| | RTX 3060 (author) | RTX 2080 Ti (us) |
|---|---|---|
| FP16 TFLOPS | ~12.7 | **~26.9** (2.1× faster) |
| PCIe bandwidth | Gen4 ≈ 24 GB/s | Gen3 ≈ 12 GB/s (2× slower) |
| Upload time (176 MiB) | ~7 ms | ~14 ms |
| Compute per MoE layer | ~18 ms | ~9 ms |
| **Upload:Compute ratio** | **0.4:1** ← overlap works | **1.5:1** ← upload dominates |

Prefetch overlap helps when compute takes longer than upload. On the 3060, slower GPU gives uploads ~11ms of slack per layer. On the 2080 Ti, the faster GPU finishes compute before upload completes — the "overlap" degrades to sequential execution plus stream/event overhead. The card is too fast at compute for PCIe Gen3 prefetch to benefit.

**Author's results (RTX 3060, sm_86, PCIe Gen4, without --no-mmap):** 1143 → 1880 t/s (+64.5%). Their gain came from combining pin-mmap (equivalent to our `--no-mmap`) + overlap. On our hardware, the pin-mmap equivalent is already active; overlap adds only overhead.

## Usage

```bash
# Opt-in via env var (NOT recommended on RTX 2080 Ti)
GGML_SCHED_PREFETCH_EXPERTS=1 llama-server-sm75 ...

# Slot count override (default 3)
GGML_SCHED_PREFETCH_EXPERTS=4 llama-server-sm75 ...

# Disable (default, recommended)
llama-server-sm75 ...  # no env var = no prefetch
```

## Pitfalls

1. **Base commit matters.** The patches must be applied on a commit that supports the target model architecture. Our first attempt was on 048a490 (731 commits behind c576070) which lacked SSM support for qwen35moe — model load failed with `missing tensor 'blk.40.ssm_conv1d.weight'`.
2. **`rm -rf build` is mandatory after `git reset --hard`.** CMake doesn't detect changed CUDA template instantiations across different base commits. Incremental builds produce linker errors (`undefined reference to ggml_cuda_flash_attn_ext_*`).
3. **The prefetch allocates ~3× max expert tensor size in VRAM** for staging slots. On 11GB, this is negligible (~200-400 MiB) but matters on tighter cards.
4. **Not upstreamable.** llama.cpp's CONTRIBUTING.md bans AI-generated PRs. These patches are fork-only.