# Qwen3.6-27B DeltaNet Architecture

The Qwen3.6-27B is a **dense 27B** model using a novel **Gated DeltaNet + Gated Attention** hybrid architecture. This is distinct from the MoE architecture used by Qwen3.6-35B-A3B.

## Architecture

From the official model card:

- **Type:** Causal Language Model with Vision Encoder (mmproj)
- **Params:** 27B (dense — all 27B active per token)
- **Layers:** 64 (non-MTP) / 65 (MTP variant)
- **Hidden dim:** 5120
- **Context:** 262,144 natively (256K), extensible to 1M+
- **Architecture pattern:** `16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))`
- **SSM config:** conv_kernel=4, state_size=128, group_count=16, dt_rank=48, inner_size=6144
- **Special:** Trained with MTP (multi-token prediction) output heads

### Layer Structure

Every 4 layers, 3 use DeltaNet (with SSM + gated attention) and 1 uses standard attention:

- **DeltaNet layers (position 0,1,2 mod 4):** ssm_conv1d, ssm_a/alpha/beta/dt/norm/out, attn_qkv, attn_gate, ffn_gate/down/up
- **Attention layers (position 3 mod 4):** attn_q/k/v (separate), attn_q_norm/k_norm, attn_output, ffn_gate/down/up

## VRAM and Performance on GTX 1080 Ti (11 GB)

| Quant | File Size | BPW | GPU Fit | Gen Speed | Notes |
|-------|-----------|-----|---------|-----------|-------|
| Q3_K_M | 12.64 GB | 4.04 | Partial (-ngl ~35) | ~15-25 t/s (est.) | Best quality/speed balance |
| UD-Q3_K_XL | 13.76 GB | 4.33 | Partial (-ngl ~30) | ~12-20 t/s (est.) | Higher quality, tighter VRAM |
| Q4_K_M | ~15 GB | ~4.5 | Too large | — | Wouldn't fit even partially well |

**Key insight:** Even at Q3/K_M (4.04 BPW), all 27B parameters are active per token — ~9x more compute than Qwen3.6-35B-A3B MoE's 3B active. This means the 27B dense can be smarter for complex reasoning despite lower quantization.

## MTP (Multi-Token Prediction) Variant — Known Limitation

The MTP GGUF variant (`unsloth/Qwen3.6-27B-MTP-GGUF`) has **65 layers** (64 main + 1 MTP head layer). Upstream llama.cpp's `qwen35.cpp` model loader has a bug handling this:

1. **`recurrent_layer_arr` miscalculation:** The loop `((i + 1) % full_attn_interval != 0)` applies to all 65 layers, but layer 64 (MTP head) is attention-only and has no SSM tensors. This causes `missing tensor 'blk.64.ssm_conv1d.weight'`.

2. **Missing `nextn.*` tensor loading:** The MTP head's 4 extra tensors (`nextn.eh_proj`, `nextn.enorm`, `nextn.hnorm`, `nextn.shared_head_norm`) aren't created by the `qwen35.cpp` loader.

**Fix approach** (if you need MTP support):
1. In `src/models/qwen35.cpp` `load_arch_hparams()`: read `nextn_predict_layers` from KV metadata, subtract from effective layer count for `recurrent_layer_arr`, and mark MTP layers as non-recurrent:
   ```cpp
   uint32_t nextn_predict_layers = 0;
   ml.get_key(LLM_KV_NEXTN_PREDICT_LAYERS, nextn_predict_layers, false);
   uint32_t effective_layers = hparams.n_layer - nextn_predict_layers;
   for (uint32_t i = 0; i < effective_layers; ++i) {
       hparams.recurrent_layer_arr[i] = ((i + 1) % full_attn_interval != 0);
   }
   for (uint32_t i = effective_layers; i < hparams.n_layer; ++i) {
       hparams.recurrent_layer_arr[i] = false;
   }
   ```
2. In `src/models/qwen35.cpp` `load_arch_tensors()`: add nextn tensor loading with `TENSOR_NOT_REQUIRED`:
   ```cpp
   layer.nextn.eh_proj          = create_tensor(tn(LLM_TENSOR_NEXTN_EH_PROJ, "weight", i), { 2 * n_embd, n_embd }, TENSOR_NOT_REQUIRED);
   layer.nextn.enorm            = create_tensor(tn(LLM_TENSOR_NEXTN_ENORM,   "weight", i), { n_embd }, TENSOR_NOT_REQUIRED);
   layer.nextn.hnorm            = create_tensor(tn(LLM_TENSOR_NEXTN_HNORM,   "weight", i), { n_embd }, TENSOR_NOT_REQUIRED);
   layer.nextn.shared_head_norm = create_tensor(tn(LLM_TENSOR_NEXTN_SHARED_HEAD_NORM, "weight", i), { n_embd }, TENSOR_NOT_REQUIRED);
   ```
3. Rebuild.

**Already fixed in:** The Stormrage34 fork (`github.com/Stormrage34/llama.cpp-turboquant-hip`). See [references/turboquant.md](references/turboquant.md) for build instructions, merge pitfalls, and flag recommendations.

**Simpler: use the non-MTP variant** (`unsloth/Qwen3.6-27B-GGUF`) which has 64 standard layers and loads cleanly with any recent llama.cpp.

## Recommended llama-server Flags

```bash
/usr/local/bin/llama-server-cuda \
  -m /models/llm/qwen27b/Qwen3.6-27B-Q3_K_M.gguf \
  --port 8091 --host 0.0.0.0 \
  -ngl 35 \                    # Partial GPU offload (12.6GB > 11GB)
  --no-kv-offload \            # KV cache on CPU (62GB RAM)
  --jinja \                    # Jinja2 chat templates
  --chat-template-kwargs '{"enable_thinking":false}' \
  -c 65536 --ctx-size 65536 \  # 64K for Hermes
  --cont-batching \
  --threads 16 --threads-batch 32 \
  --temp 0.2 --top-p 0.95 --min-p 0.05 --top-k 20
```

Omit `--mmproj` for text-only mode. To enable vision, add `--mmproj /models/llm/qwen27b/mmproj-F16.gguf --no-mmproj-offload` (projector stays on CPU).

## Comparison: Qwen3.6-27B Dense vs Qwen3.6-35B-A3B MoE vs Gemma 4 26B-A4B

| | Qwen3.6-27B (Q3_K_M) | Qwen3.6-35B-A3B (Q4) | Gemma 4 26B-A4B (Q4) |
|---|---|---|---|
| **Type** | Dense 27B | MoE 3.6B active | MoE 4B active |
| **Active params/tok** | **27B** | 3B | 4B |
| **VRAM** | ~8.5 GB (partial) | ~10 GB | ~10.3 GB |
| **Gen speed** | ~15-25 t/s (est.) | ~12 t/s | ~24 t/s |
| **Context** | 256K native | 128K native | 256K native |
| **Vision** | ✅ (mmproj) | ❌ | ✅ (mmproj on CPU) |
| **Best for** | Deep reasoning, coding | General daily driver | Vision, speed |