# Nex-N2-mini — Onboarding Reference

**Model:** nex-agi/Nex-N2-mini (Qwen3.5-35B-A3B-Base)
**GGUF:** bartowski/nex-agi_Nex-N2-mini-GGUF (Q4_K_M, 19.92 GB)
**Download:** `huggingface.co/bartowski/nex-agi_Nex-N2-mini-GGUF/resolve/main/nex-agi_Nex-N2-mini-Q4_K_M.gguf`

## Architecture

- **Parameters:** 35B total, ~3B active per token
- **Architecture:** `qwen35moe` — 256 experts, 8 active per token
- **Hybrid layers:** 30 linear-attention (Mamba/SSM) + 10 full-attention over 40 layers
- **KV heads:** 2 (GQA, head_dim=128) — tiny KV cache: ~20 KB/token
- **Context:** 262,144 tokens (256K) native
- **Multimodal:** Has a vision encoder (27-layer ViT, 1152 hidden) — mmproj available from sjakek repo
- **MTP:** Config has `mtp_num_hidden_layers: 1`, but community GGUFs drop the `nextn_predict_layers` metadata key. MTP tensors survive in the GGUF but won't activate without patching.
- **License:** Apache 2.0
- **Recommended sampling:** temp 0.7, top_p 0.95, top_k 40 (from Nex-AGI docs)

## Fleet Config

**Binary:** `/usr/local/bin/llama-server-upstream` (ac4cdde, supports `qwen35moe`)
**llama-swap entry:**

```yaml
nex-n2-mini:
  cmd: /usr/local/bin/llama-server-upstream -m /models/downloads/nex-agi_Nex-N2-mini-Q4_K_M.gguf --port ${PORT} --host 127.0.0.1 --jinja --chat-template-kwargs '{"enable_thinking":false}' -ngl 99 --no-mmap --mlock --prio 2 --poll 80 --cont-batching --timeout 300 --threads 16 --threads-batch 32 --parallel 1 -c 250000 --ctx-size 250000 --batch-size 2048 --ubatch-size 512 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --no-host -fitt 512 --n-cpu-moe 128 --no-kv-offload --temp 0.7 --top-p 0.95 --top-k 40 --metrics
  aliases:
    - "nex-n2"
    - "nex-n2-mini"
    - "Nex-N2-mini"
    - "Nex-N2-mini-Q4_K_M"
  ttl: 1800
  concurrency: 1
```

## Performance (RTX 2080 Ti 11GB)

**Warm, sustained:** ~19-20 t/s generation, ~90-140 t/s prompt eval
**Cold load:** ~3-5s

Observed speeds at 250K ctx, `--no-kv-offload`:

| Config | PE t/s | Gen t/s | Sust t/s |
|---|---|---|---|
| Baseline (128K, 1024/256, moe128) | 121.7 | 19.0 | 19.2 |
| 200K ctx | 122.2 | 19.0 | 19.1 |
| 250K ctx | 121.7 | 19.0 | 19.2 |
| **Batch 2048/512** | **142.8** | 18.5 | 18.7 |
| Batch 512/128 | 105.7 | 19.0 | 19.2 |
| n-cpu-moe 256 | 123.2 | 18.8 | 18.9 |
| n-cpu-moe 64 | 123.8 | 18.7 | 18.8 |
| no-cont-batch | 121.7 | 18.5 | 18.6 |
| threads 8/16 | 115.6 | 18.0 | 18.1 |

**Key takeaways:**
- Context size has NO impact on speed with `--no-kv-offload` (250K same as 128K)
- Batch 2048/512 gives +17% prompt eval speed, generation unchanged
- MoE thread count (64-256) doesn't matter for this model
- 16 threads minimum; dropping to 8 hurts both PE and gen by ~5%
- turbo3/turbo4 cache types NOT supported by upstream build
- `--flash-attn off` + `--cache-type-v q8_0` causes startup error (V cache needs FA)

## MTP Investigation

The original HF config has `mtp_num_hidden_layers: 1`. The sm75/MTP-Turbo build detects MTP tensors and overrides architecture from `qwen35moe` → `qwen35moe_mtp`, but crashes with:

```
GGML_ASSERT(hparams.nextn_predict_layers > 0 && "QWEN35MOE_MTP requires nextn_predict_layers > 0")
```

The GGUF metadata key `qwen35moe.nextn_predict_layers` is missing (value 0). Community quantizer (bartowski) didn't carry over `mtp_num_hidden_layers` as the correct GGUF key. The MTP weight tensors ARE in the GGUF (733 tensors vs ~700 for non-MTP variant). Adding the key would require rewriting the GGUF header — not worth the effort for 19 t/s.

No pre-built GGUF with MTP exists. Only MLX format MTP (Apple Silicon) is available from `jedisct1/Nex-N2-mini-mlx-OptiQ-4bit-MTP`.

## Context Size Calculation

With `--no-kv-offload`, KV cache lives in system RAM:
- KV cache: 2 heads × 128 dim × seq_len × 40 layers × 2 bytes = 20,480 × seq_len bytes
- 128K: ~2.6 GB RAM
- 250K: ~5.1 GB RAM
- 256K: ~5.2 GB RAM