## ⚡ Each donation = another big MoE quantized

I host **30+ free APEX MoE quantizations** as independent research. My only local hardware is an **NVIDIA DGX Spark** (122 GB unified memory) — enough for ~30-50B-class MoEs, but **bigger ones (200B+) require rented compute** on H100/H200/Blackwell, typically $20-100 per quant.

If APEX quants are useful to you, your support directly funds those bigger runs.

[🎉 Patreon (Monthly)](https://www.patreon.com/cw/mudler)  \|
[☕ Buy Me a Coffee](https://www.buymeacoffee.com/mudler)  \|
[⭐ GitHub Sponsors](https://github.com/sponsors/mudler)

# Qwen3.6-35B-A3B — APEX-MTP GGUF

**APEX (Adaptive Precision for EXpert Models)** quantizations of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B), with the **MTP (multi-token prediction) head bundled** for in-the-box self-speculative decoding.

**Brought to you by the [LocalAI](https://github.com/mudler/LocalAI) team** \| [APEX Project](https://github.com/mudler/apex-quant) \| [Technical Report](https://github.com/mudler/apex-quant/blob/main/paper/APEX_Technical_Report.pdf)

## What's different from the plain APEX repo?

These GGUFs bundle the model's **MTP (multi-token prediction) head** alongside the trunk in a single file, courtesy of [llama.cpp PR #22673](https://github.com/ggml-org/llama.cpp/pull/22673). With a recent llama.cpp (>= commit 255582687) you can enable self-speculative decoding using just this one file — no separate draft model needed:

```bash
llama-server -m Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --draft-mtp
```

The non-MTP version is still available at [mudler/Qwen3.6-35B-A3B-APEX-GGUF](https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-GGUF) — slightly smaller, but no self-spec.

## File sizes

Each quant is ~2.5% larger than its non-MTP counterpart (one extra transformer-block worth of weights, no embedding duplication since MTP shares the trunk's embed\_tokens).

## MTP draft head precision

The bundled MTP head (`blk.40.*` including the `nextn.*` projection + norms) is
quantized to **Q8\_0** (near-lossless) on **every tier except I-Nano**. I-Nano keeps
the trunk-tier precision on the MTP block (Q3\_K routed experts, Q4\_K attention)
but pins `blk.40.nextn.eh_proj` to Q4\_K — see the [explainer below](https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF#why-the-mtp-head-doesnt-use-imatrix).

This keeps draft accuracy high (important for spec-decode acceptance rate) at a
modest ~1 GB cost per file vs. trunk-tier precision.

### Why the MTP head doesn't use imatrix

`llama-imatrix` runs normal forward passes that only activate the trunk
(`blk.0..blk.39`). The MTP head only fires during `--draft-mtp` spec decoding,
so its tensors get no imatrix activation data. We work around this by
quantizing the MTP head with static K-quant / Q8\_0 which doesn't require
imatrix.

(A patch to `llama-imatrix` that records MTP activations during collection
is in progress at [mudler/llama.cpp#mtp-imatrix](https://github.com/mudler/llama.cpp/tree/mtp-imatrix)
— once upstream this will let us push the drafter to lower bit-widths cleanly.)

## What is APEX?

APEX is a MoE-aware mixed-precision quantization strategy. Per-tensor-role gradient: routed experts compress hardest, shared experts kept high (always active), attention/Mamba uniform; 5+5 symmetric edge gradient across the 40 trunk layers + MTP layer 40 at edge precision. I-variants use diverse imatrix calibration (chat, code, reasoning, tool-calling, agentic traces, Wikipedia).

See the [APEX project](https://github.com/mudler/apex-quant) for full details.

## Architecture

- **Base**: Qwen 3.6 35B-A3B family (Qwen3\_5MoeForCausalLM)
- **Layers**: 40 trunk + 1 MTP (bundled)
- **Experts**: 256 routed + 1 shared (8 active per token)
- **Hidden size**: 2048
- **Calibration**: v1.3 diverse dataset

## Credits

- **APEX quantization**: [LocalAI](https://github.com/mudler/LocalAI) team
- **MTP support**: llama.cpp PR #22673 by Aman Gupta + ggerganov
- Built on [llama.cpp](https://github.com/ggerganov/llama.cpp)

Downloads last month13,205

GGUF

Model size

36B params

Architecture

qwen35moe

Chat template

Hardware compatibility

[Log In](https://huggingface.co/login?next=https%3A%2F%2Fhuggingface.co%2Fmudler%2FQwen3.6-35B-A3B-APEX-MTP-GGUF) to add your hardware

16-bit

F16

71.1 GB

[View +8 variants](https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/tree/main)

Inference Providers [NEW](https://huggingface.co/docs/inference-providers)

This model isn't deployed by any Inference Provider. [🙋Ask for provider support](https://huggingface.co/spaces/huggingface/InferenceSupport/discussions/new?title=mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF&description=React%20to%20this%20comment%20with%20an%20emoji%20to%20vote%20for%20%5Bmudler%2FQwen3.6-35B-A3B-APEX-MTP-GGUF%5D(%2Fmudler%2FQwen3.6-35B-A3B-APEX-MTP-GGUF)%20to%20be%20supported%20by%20Inference%20Providers.%0A%0A(optional)%20Which%20providers%20are%20you%20interested%20in%3F%20(Novita%2C%20Hyperbolic%2C%20Together%E2%80%A6)%0A)

## Model tree for mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF

Base model

[Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)

Quantized

( [565](https://huggingface.co/models?other=base_model:quantized:Qwen/Qwen3.6-35B-A3B))

this model

## Collection including mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF

[MoE models quantized with the APEX Quantization technique ( https://github.com/mudler/apex-quant )•39 items•Updated 1 day ago• 124](https://huggingface.co/collections/mudler/apex-quants-gguf)

Inference providers allow you to run inference using different serverless providers.

StripeM-Inner