# `--n-cpu-moe` Source-Verified Behavior

Pulled from `common/arg.cpp` and `common/common.h` in llama.cpp:

## Regex Target

```
LLM_FFN_EXPS_REGEX = "\.ffn_(up|down|gate|gate_up)_(ch|)exps"
```

Only MoE FFN expert weight tensors (up/down/gate/gate_up projections) are matched. Attention weights, layer norms, embeddings, and output layer are NOT affected.

## What `--n-cpu-moe N` Actually Does

For each layer `i` from `0` to `N-1`, creates a `tensor_buft_override`:
```
blk.i.ffn_up_exps     → ggml_backend_cpu_buffer_type()
blk.i.ffn_down_exps   → ggml_backend_cpu_buffer_type()
blk.i.ffn_gate_exps   → ggml_backend_cpu_buffer_type()
```

These overrides take priority over `-ngl`. Even with `-ngl 99`, matched tensors go to CPU.

### What stays on GPU (with `-ngl 99`)
- Attention weights (QKV/O projections)
- Layer norms, embeddings, output/head layer
- Shared experts (not matched by regex — they use different tensor names)
- Expert weights in layers N through top (no override created for them)

## CRITICAL: tensor_buft_overrides Disables Auto-Fit

When `--n-cpu-moe` or `--cache-type-k/v` is set, `params.tensor_buft_overrides` is populated. This **blocks the auto-fitting algorithm** in `common_fit_params`. The log shows:

```
W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user, abort
```

Without auto-fit, `-ngl 99` forces EVERY non-overridden tensor to GPU regardless of VRAM. If they don't fit, you get:

```
E ggml_backend_cuda_buffer_type_alloc_buffer: allocating X MiB on device 0: cudaMalloc failed: out of memory
```

**Rule:** When using `--n-cpu-moe` or `--cache-type-k/v`, you MUST explicitly set `-ngl` to a value that fits your VRAM. Auto-fit will not save you.

## Two Variants

| Flag | Source code effect |
|------|--------------------|
| `--cpu-moe` / `-cmoe` | Creates ONE override targeting ALL layers at once via the bare regex |
| `--n-cpu-moe N` / `-ncmoe N` | Creates N overrides, one per layer index 0..N-1 |

## Detection / Recovery Pattern

When `--n-cpu-moe + --cache-type-k/v + --flash-attn + -ngl 99` cause unexpected OOM:
1. Remove all four flags
2. Start without `-ngl` — let auto-fit pick the right number of GPU layers
3. Note what `-ngl` auto-fit chose (check stderr)
4. Re-add flags with that explicit `-ngl` value

## Models Requiring This Fix

| Model | Why needed | Optimal `--n-cpu-moe` |
|-------|-----------|----------------------|
| DeepSeek-Coder-V2-Lite Q5_K_M (11.85 GB) | Expert weights ~65% of file, OOMs at any ctx without offload | 16 (27 layers total) |
| Gemma 4 26B A4B Q4_K_XL (15.6 GB) | File > VRAM, experts are CPU-bound anyway | 16 (30 layers total) |
| Nemotron-3-Nano-30B Q4_K_M (23 GB) | Massive file needs most experts on CPU | 128 (52 layers total) |
| **Qwen3-Coder-Next Q4_K_M (46 GB)** | **46 GB file, 48 layers. n-cpu-moe 42 leaves 6 layers' MoE on GPU = ~5 GB → OOM on 11GB card. Must offload ALL experts.** | **48** (all MoE on CPU) |