# MoE Offloading Research & GGUF Parsing Findings (2026-06-22)

## GGUF v3 Metadata Extraction

When using Python to read GGUF headers from large model files (>5 GB), the `gguf.GGUFReader` library times out. Extract metadata directly from raw bytes.

**Key finding:** GGUF v3 uses **u64 (8 bytes)** for key length, NOT u32 (4 bytes) as in v2. Using u32 shifts all subsequent reads by 4 bytes.

```python
# Correct v3 parsing:
kl = struct.unpack('<Q', data[pos:pos+8])[0]  # u64 key length
key = data[pos+8:pos+8+kl].decode('utf-8')
pos += 8 + kl
vt = struct.unpack('<I', data[pos:pos+4])[0]  # u32 value type
```

**Workflow:**
1. Read first 2MB of the GGUF file with `head -c 2097152`
2. Parse KV pairs from the buffer
3. Skip array values (tokenizer vocab) by reading array type + length and advancing pos
4. Target specific keys: `general.architecture`, `<arch>.context_length`, `general.file_type`, etc.

## Model Research Sources

### GLM-4.7-Flash
- **Base**: zai-org/GLM-4.7-Flash (30B-A3B MoE, deepseek2 arch)
- **GGUF**: unsloth/GLM-4.7-Flash-GGUF
- **Native ctx**: 202K
- **Key flags**: `--reasoning off` REQUIRED. GLM natively outputs `reasoning_content` without it.
- **Z.ai recommended sampling**: temp 1.0, top-p 0.95, min-p 0.01, repeat-penalty 1.0
- **DeepSeek v2/v3 arch caveat**: May need stripped flags (no --prio, --mlock, --cont-batching, --no-host -fitt) if tensor_buft_overrides conflict occurs
- **Benchmarks**: Leads Qwen3-30B-Thinking and GPT-OSS-20B on most agentic/coding benchmarks (AIME 25: 91.6, SWE-bench: 59.2)
- **MTP support**: Some reports mention "embedded draft model" — test with `--spec-type draft-mtp`

### GPT-OSS-20B
- **Base**: openai/gpt-oss-20b (20.9B total, 3.6B active MoE)
- **GGUF**: unsloth/GPT-OSS-20B-GGUF
- **Arch**: gpt-oss, 32 experts, 24 layers, 64 query heads, 8 KV heads
- **Hidden dim**: 2880, residual dim: 2880
- **Native ctx**: 131K (YaRN)
- **Native quantization**: MXFP4 (4.25 bits/param) on MoE weights
- **MoE**: NOT dense! Easy mistake. Active params only 3.6B per token.
- **Reddit report**: "gpt-oss-20b fully in VRAM and get ~200 t/s"
- **Q6_K file**: 12 GB (smaller than expected due to MXFP4 heritage)
- **License**: Apache 2.0

### Qwen3.6-35B-A3B-MTP
- **Base**: Qwen/Qwen3.6-35B-A3B
- **GGUF**: unsloth/Qwen3.6-35B-A3B-GGUF
- **Arch**: qwen35moe, 41 layers, 16/2 heads, 262K ctx
- **Native MTP**: has built-in MTP head for spec decoding
- **Opt ncm**: 32 (validated: 60 t/s @ 9.2 GB)

### Gemma 4 26B A4B QAT
- **GGUF**: yuxinlu1/gemma4-v2
- **Arch**: gemma4, 30 layers, 16 heads, 262K ctx
- **Draft**: mtp-gemma-4-26B-A4B-it.gguf (423M, 4 layers)
- **Note**: `gemma4` arch (not `gemma4_unified`) — requires sm75 binary
- **--no-kv-offload**: Keeps KV cache on CPU — 230K ctx costs ZERO VRAM

## MoE Offloading Principles

From HuggingFace blog (Doctor-Shotgun, Jan 2026) and DEV community analysis:

1. **`--n-cpu-moe N`**: Moves MoE expert weights of the first N layers to CPU
2. **How it works**: The first N layers' `blk.N.ffn_*_exps*` tensors are assigned to CPU; attention/dense weights of ALL layers stay on GPU
3. **Default 128**: All experts on CPU (safe, low VRAM). Lower N → more experts on GPU → faster but more VRAM
4. **No formula for optimal** — trial and error per model/hardware combo
5. **Bottleneck**: PCIe latency, not CPU speed. Each token activation crosses PCIe per offloaded layer
6. **Smaller files benefit most** (15-25 GB range) — moving significant fraction of experts to GPU changes the balance
7. **Largest MoE files (45+ GB)** gain minimally — even moving 6/48 experts barely affects PCIe bottleneck
8. **Combined with `--no-kv-offload`**: Frees 2-5 GB VRAM for model weights instead of KV cache

### VRAM Budget Calculation (11GB RTX 2080 Ti)

For any new MoE model:
1. Read file size and layer count from GGUF metadata
2. Estimate expert weight per layer: `file_size / num_layers * 0.6`
3. Start ncm at 128, drop by 5 until VRAM approaches 10.5 GB
4. Add ~1 GB for compute buffer overhead
5. If KV cache exceeds 2 GB, enable `--no-kv-offload`
6. Verify with actual load test — benchmark at found ncm

### DeepSeek v2/v3 Architecture

Models using deepseek2 arch (GLM-4.7-Flash, DeepSeek V2/V3, Moonlight):
- May have `tensor_buft_overrides` conflicts with --cache-type-k/v, --mlock, --prio, --cont-batching
- If model crashes at load: strip flags to minimum (no --mlock, no --prio, no --cont-batching, no --no-host -fitt X)
- Reintroduce flags one at a time
- --reasoning off behavior: GLM models fully obey this flag; Qwen still emits thinking tags regardless

### Context / VRAM Tradeoffs

- `-c` (context length) linearly affects KV cache size
- With `--no-kv-offload`: context increases have ZERO effect on GPU VRAM
- With GPU KV cache: increasing ctx by 10% increases VRAM by ~2-5% depending on model
- Compute graph buffer grows with ctx for full-attention layers: ~477 MB at 128K, ~708 MB at 262K on Qwen architectures
- Smaller batch sizes (256/64 vs 2048/512) save ~1 GB compute buffer with no gen-speed impact