# Qwen & Gemma Model Specs (from official Hugging Face model cards)

Verified 2025-06-04. Source: Official HF README.md for each model.

## Qwen3.6-35B-A3B

- **Architecture**: 40 layers, MoE (sparse). Hybrid Gated DeltaNet + Gated Attention.
- **Layout**: 10 × (3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE))
- **Context Length**: **262,144 tokens** natively — extensible up to **1,010,000 tokens**.
- **Vocabulary**: 248,320 tokens (padded)
- **Pipeline tag**: `image-text-to-text`
- **License**: Apache 2.0
- **Base model**: `Qwen/Qwen3.6-35B-A3B`
- **MTP support**: Yes (Unsloth MTP GGUF variant). See `unsloth.ai/docs/models/qwen3.6#mtp-guide`
- **Unsloth MTP GGUF**: `unsloth/Qwen3.6-35B-A3B-MTP-GGUF`
- **Recommended for**: Coding, reasoning, agentic workflows. Top fleet performer for complex tasks.

## Qwen3.5-9B

- **Architecture**: 32 layers, Gated DeltaNet + FFN. Hybrid: 8 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))
- **Context Length**: **262,144 tokens** natively — extensible up to **1,010,000 tokens**.
- **Warning from Qwen docs**: If OOM with 262K, reduce. "Maintain at least 128K to preserve thinking capabilities."
- **Vocabulary**: 248,320 tokens (padded)
- **License**: Apache 2.0
- **MTP support**: Yes (Unsloth MTP GGUF variant). `unsloth/Qwen3.5-9B-MTP-GGUF`
- **Note**: Qwopus3.5-9B-Coder is a fine-tune of this base model — inherits all architecture specs.

## Gemma 4 26B A4B MoE

- **Architecture**: 30 layers, MoE. 8 active / 128 total experts + 1 shared expert.
- **Active Parameters**: 3.8B
- **Total Parameters**: 25.2B
- **Context Length**: **256K tokens** (medium model category per Google)
- **Sliding Window**: 1024 tokens
- **Vocabulary**: 262,000 tokens
- **Supported Modalities**: Text, Image
- **Vision Encoder**: ~550M parameters (external encoder)
- **License**: Apache 2.0 (Gemma 4 license)
- **MTP support**: NO. Gemma 4 architecture has NO MTP support in llama.cpp. PR #24057 was abandoned.
- **Thinking mode**: Native. Control with `<|think|>` token. Disable with `--reasoning off`.
- **Recommended sampling (Google)**: temp=1.0, top_p=0.95, top_k=64. For agent use: temp=0.2.

## Gemma 4 12B Unified Dense

- **Architecture**: 48 layers, dense (NOT MoE). Encoder-free (unified) — projects raw images/audio via linear layers, no external encoder.
- **Total Parameters**: 11.95B
- **Context Length**: **256K tokens**
- **Sliding Window**: 1024 tokens
- **Vocabulary**: 262,000 tokens
- **Supported Modalities**: Text, Image, Audio (ASR + speech-to-text translation)
- **License**: Apache 2.0 (Gemma 4 license)
- **MTP support**: NO. See Gemma 4 26B entry.
- **Thinking mode**: Native. Controlled via `<|think|>` token in system prompt.
- **Recommended sampling (Google)**: temp=1.0, top_p=0.95, top_k=64.
- **Quantization on 2080 Ti**: Q4_K_M (4.77 BPW, 6.62 GiB file). Q6_K too large at ~9.5 GiB.
- **Variable image resolution**: Supports token budgets 70/140/280/560/1120. Place image BEFORE text.
- **Video**: Up to 60 seconds. Audio: up to 30 seconds.