# Model Architecture Reference — 2026-06-19 Fleet

## Nemotron-3-Nano-30B-A3B (unsloth GGUF, Q4_K_M)

| Property | Value |
|----------|-------|
| **Architecture** | `nemotron_h` (NVIDIA) |
| **Type** | MoE: 30B total / 3B active per token |
| **Layers** | 52 |
| **KV heads** | **2** — tied with Qwen3-Coder-Next for most aggressive GQA (8:1) |
| **Max context** | 131K tested, architecture supports more |
| **Binary** | upstream only |
| **MoE offload** | `--n-cpu-moe 128` (experts CPU, non-experts GPU) |
| **Benchmark** | 36.4 t/s, 3.4 GB VRAM at 128K |
| **File size** | 23 GB (Q4_K_M) |
| **Download** | `hf download unsloth/Nemotron-3-Nano-30B-A3B-GGUF --include "*Q4_K_M*" --local-dir /models/downloads/` |
| **Notes** | 2 KV heads make KV cache essentially free. Can push to 256K at ~5 GB total VRAM. Cold start 35s. |

## Nemotron-Terminal-14B (mradermacher GGUF, Q4_K_S)

| Property | Value |
|----------|-------|
| **Architecture** | Qwen3 (gated_delta_net + gated_attention hybrid) |
| **Type** | Dense 14B |
| **Layers** | 40 |
| **KV heads** | 8 (GQA) |
| **Binary** | upstream (no MTP — GGUF has no MTP heads) |
| **Flag** | `--top-k 40` (Qwen3 standard) |
| **Benchmark** | 56.2 t/s, 10.6 GB VRAM at 32K |
| **File size** | 8.0 GB (Q4_K_S) |
| **Download** | `hf download mradermacher/Nemotron-Terminal-14B-GGUF --include "*Q4_K_S*" --local-dir /models/downloads/` |
| **Notes** | VRAM at 32K is 10.6 GB / 11 GB = 96%. 64K OOMs. |

## DeepSeek-Coder-V2-Lite-Instruct (bartowski GGUF, Q5_K_M)

| Property | Value |
|----------|-------|
| **Architecture** | `deepseek_v2` (DeepSeek) |
| **Type** | MoE: ~16B total / 2.4B active. Top-k 6 + 2 shared experts. |
| **Layers** | 27 |
| **Attention** | **MLA** (Multi-head Latent Attention): KV compression rank 512, QK rope dim 64 |
| **KV cache size** | ~15,552 bytes/token at q8_0 (27 × 576 values) — tiny |
| **Max context** | 163,840 native |
| **Head dim** | 128 (16 heads) |
| **Binary** | upstream ONLY (sm75 doesn't support deepseek_v2) |
| **Prompt format** | DeepSeek: `<｜begin▁of▁sentence｜>User: ...\n\nAssistant: <｜end▁of▁sentence｜>Assistant:` |
| **MoE offload** | `--n-cpu-moe 16` — first 16 layers' experts on CPU. Layers 16-26 experts + attention + non-expert weights on GPU. |
| **VRAM at 32K q8_0** | 10.2 GB (Q5_K_M), 33.9 t/s |
| **Critical flag rules** | `--n-cpu-moe` + `--cache-type-k/v` + `--flash-attn` sets `tensor_buft_overrides` → blocks auto-fit. Must explicitly set `-ngl 99` and `-c 32768`. Cannot add `--mlock`, `--prio`, `--cont-batching` or OOM. |
| **File size** | 11.85 GB (Q5_K_M) |
| **Download** | `hf download bartowski/DeepSeek-Coder-V2-Lite-Instruct-GGUF --include "*Q5_K_M*" --local-dir /models/downloads/` |

## Hugging Face CLI Migration

`huggingface-cli` is **deprecated** as of June 2026. Use the new `hf` CLI:

```bash
# OLD (broken)
huggingface-cli download USER/REPO --include "file.gguf" --local-dir /models/

# NEW (working)
hf download USER/REPO --include "file.gguf" --local-dir /models/
```

`hf` comes pre-installed with the `pipx` huggingface-hub package. Run `hf --help` for all commands.

---

## Qwen3-Coder-Next (unsloth GGUF, Q4_K_M)

| Property | Value |
|----------|-------|
| **Architecture** | Hybrid: Gated DeltaNet + Gated Attention + MoE |
| **Type** | MoE: 80B total / 3B active per token |
| **Layers** | 48 (12×[3×DeltaNet→MoE + 1×Attention→MoE]) |
| **KV heads** | 2 (GQA 8:1 ratio, 16 Q heads, **head_dim 256**) |
| **Max context** | 256K native |
| **Binary** | sm75 (llama-server-sm75, b9743) |
| **MoE offload** | `--n-cpu-moe 48` — **REQUIRED** on 11GB card. 42 leaves 6 layers on GPU → OOM |
| **VRAM at 32K q8_0** | ~9.6-10.6 GB (fits, tight). 64K ctx → OOM |
| **KV cache** | 48 KB/token at q8_0 (all 48 layers allocated, 12 actually produce KV) |
| **File size** | 46 GB (Q4_K_M, 48.5 GB nominal) |
| **Download** | `hf download unsloth/Qwen3-Coder-Next-GGUF --include "*Q4_K_M*" --local-dir /models/downloads/` |
| **Notes** | 512 experts, 10 active + 1 shared. Non-thinking only (no `<think>` blocks). ~~`--n-cpu-moe 42` caused OOM at 128K ctx~~ → Validated: `--n-cpu-moe 46` at 128K ctx = **33.7 t/s**, no OOM. Non-expert weights on GPU = ~0.8 GB (not 8-9 GB — the effective weight is much smaller because non-expert params are only ~1.5B of 80B total). KV cache at 128K/q8_0 = 6.29 GB. Total VRAM ~8.9 GB, fits on 11 GB card. |