# DeepSeek v2/v3 / Moonlight Architecture Flags

**Updated:** 2026-06-19  
**GPU:** RTX 2080 Ti 11GB (SM75 Turing)  
**Build:** upstream (ac4cdde)

## Architecture Overview

| Model | Arch Type | Layers | Hidden | Heads | KV | MoE Top-K | MLA | Context |
|---|---|---|---|---|---|---|---|---|
| DeepSeek-Coder-V2-Lite | `deepseek_v2` | 27 | 2048 | 16 | 16 | 6 | ✅ (rank 512) | 163K |
| Moonlight-16B-A3B | `deepseek_v3` | 27 | 2048 | 16 | 16 | 6 | ✅ | 8K |

Both use MLA (Multi-head Latent Attention) — KV compression rank 512 + QK rope dim 64.
KV cache per layer = kv_lora_rank + qk_rope_head_dim = 512 + 64 = 576 values.

## CRITICAL: `tensor_buft_overrides` Conflict

### Source of the Problem

`--n-cpu-moe` (and `--cpu-moe`) register `tensor_buft_overrides` in `common/arg.cpp`:

```
"--n-cpu-moe" → for each i in 0..N-1:
    params.tensor_buft_overrides.push_back({blk_regex(i), ggml_backend_cpu_buffer_type()})
```

The regex is defined in `common/common.h`:
```
LLM_FFN_EXPS_REGEX = "\.ffn_(up|down|gate|gate_up)_(ch|)exps"
```

This matches only MoE FFN expert weight tensors in layers 0..N-1 and forces them to CPU memory.

### Three-Way Conflict

1. `--n-cpu-moe` sets `tensor_buft_overrides` → **blocks auto-fit**
2. `--cache-type-k/v` also sets `tensor_buft_overrides` → cannot be combined with (1)
3. `--mlock`, `--prio`, `--cont-batching`, `--no-host -fitt 512`, `--batch-size` may also trigger the override check

When ANY of these is set, the auto-fitting algorithm emits:
```
model_params::tensor_buft_overrides already set by user, abort
```
and falls through to the forced `-ngl 99` allocation, which OOMs on models > 11 GB.

### Configs That Work

**Option A: Fastest (with --n-cpu-moe, stripped flags)**
```
/usr/local/bin/llama-server-upstream
  -m <model.gguf>
  --jinja --reasoning off -ngl 99 --no-mmap
  -c 32768
  --n-cpu-moe 16
  --cache-type-k q8_0 --cache-type-v q8_0
  --flash-attn on
  --temp 0.2 --top-p 0.95 --min-p 0.05
  --metrics
```
- 33.9 t/s, 10.2 GB VRAM on DeepSeek-Coder-V2-Lite Q5_K_M
- ~800 MB headroom — fragile
- NO: --mlock, --prio, --cont-batching, --no-host -fitt 512, --threads-batch

**Option B: Most Stable (no --n-cpu-moe, let auto-fit decide)**
```
/usr/local/bin/llama-server-upstream
  -m <model.gguf>
  --jinja --reasoning off --no-mmap --mlock
  --prio 2 --poll 30 --no-cont-batching --timeout 300
  --threads 16 --threads-batch 32 --parallel 1
  -c 65536
  --batch-size 1024 --ubatch-size 256
  --temp 0.2 --top-p 0.95 --min-p 0.05 --top-k 20
  --metrics
```
- 24.9 t/s, 9.5 GB VRAM on DeepSeek-Coder-V2-Lite Q5_K_M
- 1.7 GB headroom — stable
- Auto-fit handles layer distribution dynamically
- ALL standard flags work

**Option C: Reduced Context (keeps speed, gains headroom)**
Same as Option A but `-c 16384`:
- 39.5 t/s, 7.9 GB VRAM, 3.1 GB headroom
- Best speed/headroom tradeoff

## Moonlight-16B-A3B-Instruct-bruno Q5_K_M

- File: 12.1 GB
- Arch: `deepseek_v3` (DeepseekV3ForCausalLM)
- Context: 8K native (max_position_embeddings=8192)
- Same flag constraints as DeepSeek-Coder-V2-Lite
- Estimated VRAM with --n-cpu-moe 16 + q8_0 cache: ~8.5 GB at 8K context (2.8 GB headroom)

## Building a New Upstream Binary

Current upstream: `ac4cdde` (very old). Latest: **b9733** (as of 2026-06-19).
A newer build may fix the tensor_buft_overrides interaction. To build:
```bash
cd ~/src/llama.cpp-upstream
git fetch --tags
git checkout b9733
rm -rf build-cuda && mkdir build-cuda && cd build-cuda
cmake .. -DGGML_CUDA=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.6/bin/nvcc \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-13 \
  -DGGML_CUDA_GRAPHS=ON
make -j$(nproc)
sudo cp bin/llama-server /usr/local/bin/llama-server-upstream
```