[Introducing Unsloth Studio: a new web UI for local AI\\
\\
🦥](https://unsloth.ai/docs/new/studio)

For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). This page is also available as [Markdown](https://unsloth.ai/docs/models/qwen3.6.md).

Qwen3.6 is Alibaba’s new family of multimodal hybrid-thinking models, including: **Qwen3.6-27B** and **35B-A3B**. It delivers top performance for its size, supports 256K context across 201 languages. It excels in agentic coding, vision, chat tasks. Qwen3.6-27B runs on **18GB RAM** setups and 35B-A3B runs on **22GB**. You can now run and train the models in [Unsloth Studio](https://unsloth.ai/docs/models/qwen3.6#unsloth-studio-guide).

**NEW:** [**Qwen3.6 MTP is here**](https://unsloth.ai/docs/models/qwen3.6#mtp-guide) **! MTP enables 1.4-2.2x faster inference without accuracy loss. Run MTP directly in** [**Unsloth Studio**](https://unsloth.ai/docs/models/qwen3.6#unsloth-studio-mtp-guide) **.**

We conducted [Qwen3.6 GGUF Benchmarks](https://unsloth.ai/docs/models/qwen3.6#unsloth-gguf-benchmarks) to help you pick the best quant.

[Run Qwen3.6 Tutorials](https://unsloth.ai/docs/models/qwen3.6#qwen3.6-inference-tutorials) [MTP Guide](https://unsloth.ai/docs/models/qwen3.6#mtp-guide)

Qwen3.6 GGUFs use Unsloth [Dynamic 2.0](https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs) for SOTA quant performance - so quants are calibrated on real world use-case datasets and important layers are upcasted. _Thank you Qwen for day zero access._

- **Developer Role Support** for Codex, OpenCode and more:Our uploads now support the `developer role` for agentic coding tools.

- **Tool calling:** Like [Qwen3.5](https://unsloth.ai/docs/models/qwen3.5), we improved parsing nested objects to make tool calling succeed more.


![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FTbH2CrUTG2TWwgOP74GF%252FGemma%25204%2520example.gif%3Falt%3Dmedia%26token%3D56409d06-3735-4531-97c0-af9968371a26&width=768&dpr=3&quality=100&sign=cf0414e5&sv=2)

Qwen3.6 running in [Unsloth Studio](https://unsloth.ai/docs/models/qwen3.6#unsloth-studio-guide).

### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#usage-guide)    ⚙️ Usage Guide

**Table: Inference hardware requirements** (units = total memory: RAM + VRAM, or unified memory)

Qwen3.6

3-bit

4-bit

6-bit

8-bit

BF16

**27B**

15 GB

18 GB

24 GB

30 GB

55 GB

**35B-A3B**

17 GB

23 GB

30 GB

38 GB

70 GB

For best performance, make sure your total available memory (VRAM + system RAM) exceeds the size of the quantized model file you’re downloading. If it doesn’t, llama.cpp can still run via SSD/HDD offloading, but inference will be slower.

Do NOT use CUDA 13.2 as you may get gibberish outputs. Use below CUDA 13.2 or CUDA 13.3.

**To train Qwen3.6, you can refer to our previous** [**Qwen3.5 fine-tuning guide**](https://unsloth.ai/docs/models/qwen3.5/fine-tune) **.**

### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#recommended-settings)    Recommended Settings

- **Maximum context window:**`262,144` (can be extended to 1M via YaRN)

- `presence_penalty = 0.0 to 2.0` default this is off, but to reduce repetitions, you can use this, however using a higher value may result in **slight decrease in performance**

- **Adequate Output Length**: `32,768` tokens for most queries


If you're getting gibberish, your context length might be set too low. Or try using `--cache-type-k bf16 --cache-type-v bf16` which might help.

As Qwen3.6 is hybrid reasoning, thinking and non-thinking mode have different settings:

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#thinking-mode)    Thinking mode:

Qwen3.6 now has [Preserve Thinking](https://unsloth.ai/docs/models/qwen3.6#turn-on-off-thinking--preserve-thinking).

General tasks

Precise coding tasks (e.g. WebDev)

temperature = 1.0

temperature = 0.6

top\_p = 0.95

top\_p = 0.95

top\_k = 20

top\_k = 20

min\_p = 0.0

min\_p = 0.0

presence\_penalty = 0.0

presence\_penalty = 0.0

repeat\_penalty = disabled or 1.0

repeat\_penalty = disabled or 1.0

Thinking mode for general tasks:

Copy

```
temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
```

Thinking mode for precise coding tasks:

Copy

```
temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
```

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#instruct-non-thinking-mode-settings)    Instruct (non-thinking) mode settings:

General tasks

temperature = 0.7

top\_p = 0.8

top\_k = 20

min\_p = 0.0

presence\_penalty = 1.5

repeat\_penalty = disabled or 1.0

To [disable thinking / reasoning](https://unsloth.ai/docs/models/qwen3.6#how-to-enable-or-disable-reasoning-and-thinking), use `--chat-template-kwargs '{"enable_thinking":false}'`

If you're on **Windows** Powershell, use: `--chat-template-kwargs "{\"enable_thinking\":false}"`

Use 'true' and 'false' interchangeably.

Instruct (non-thinking) for general tasks:

Copy

```
temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
```

## [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#qwen3.6-inference-tutorials)    Qwen3.6 Inference Tutorials:

We'll be using Dynamic 4-bit `UD-Q4_K_XL` GGUF variants for inference workloads. Click below to navigate to designated model instructions:

[Run in Unsloth Studio](https://unsloth.ai/docs/models/qwen3.5#unsloth-studio-guide) [Run in llama.cpp](https://unsloth.ai/docs/models/qwen3.6#llama.cpp-guides) [MTP Guide](https://unsloth.ai/docs/models/qwen3.6#mtp-guide) [NVFP4 Guide](https://unsloth.ai/docs/models/qwen3.6#nvfp4)

Do NOT use CUDA 13.2 as you may get gibberish outputs. Use below CUDA 13.2 or CUDA 13.3.

### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#unsloth-studio-guide)    🦥 Unsloth Studio Guide

Qwen3.6 and Qwen3.6 MTP can now be run in [Unsloth Studio](https://unsloth.ai/docs/new/studio), our new open-source web UI for local AI. Unsloth Studio lets you run models locally on **MacOS, Windows**, Linux and:

- Search, download, [run GGUFs](https://unsloth.ai/docs/new/studio#run-models-locally) and safetensor models

- [**Self-healing** tool calling](https://unsloth.ai/docs/new/studio#execute-code--heal-tool-calling) \+ **web search**

- [**Code execution**](https://unsloth.ai/docs/new/studio#run-models-locally)(Python, Bash)

- [Automatic inference](https://unsloth.ai/docs/new/studio#model-arena) parameter tuning (temp, top-p, etc.)

- Fast CPU + GPU inference via llama.cpp

- [Train LLMs](https://unsloth.ai/docs/new/studio#no-code-training) 2x faster with 70% less VRAM


![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FKoonE97b0H3RioMwVohd%252Fstudio%2520new%2520ui.gif%3Falt%3Dmedia%26token%3De37f2839-914e-48b8-8c81-2dac3ade9408&width=768&dpr=3&quality=100&sign=2c00972a&sv=2)

1

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#install-unsloth)    Install Unsloth

Run in your terminal:

**MacOS, Linux, WSL:**

Copy

```
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows PowerShell:**

Copy

```
irm https://unsloth.ai/install.ps1 | iex
```

**Installation will be quick and take approx 20 sec - 1 mins.**

2

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#launch-unsloth)    Launch Unsloth

**MacOS, Linux, WSL and Windows:**

Copy

```
unsloth studio -H 0.0.0.0 -p 8888
```

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fd1yMMNa65Ccz50Ke0E7r%252FScreenshot%25202026-03-17%2520at%252012.32.38%25E2%2580%25AFAM.png%3Falt%3Dmedia%26token%3D9369cfe7-35b1-4955-b8cb-42f7ecb43780&width=768&dpr=3&quality=100&sign=e483d141&sv=2)

Then open `http://127.0.0.1:8888` (or your specific URL) in your browser.

3

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#search-and-download-qwen3.6-or-qwen3.6-mtp)    Search and download Qwen3.6 or Qwen3.6 MTP

On first launch you will need to create a password to secure your account and sign in again later. Then go to the [Studio Chat](https://unsloth.ai/docs/new/studio/chat) tab and search for Qwen3.6 or Qwen3.6 MTP in the search bar and download your desired model and quant.

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FT6uAnOeF7OU9cuiE8JR5%252FScreenshot%25202026-04-16%2520at%25208.59.33%25E2%2580%25AFAM.png%3Falt%3Dmedia%26token%3D6977f7b6-aff7-494b-84b5-ad737125da31&width=768&dpr=3&quality=100&sign=9403a169&sv=2)

4

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#run-qwen3.6)    Run Qwen3.6

Inference parameters should be auto-set when using Unsloth Studio, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our [Unsloth Studio inference guide](https://unsloth.ai/docs/new/studio/chat). Below, the 2-bit Qwen3.6 GGUF made 30+ tool calls, searched 20 sites and executed Python code:

9lqVQm1qDX3elt6Uan5Vm.mp4

PausePlay

% buffered00:00

00:28

Exit fullscreenEnter fullscreenGo to original URL

Your browser does not support HTML5 videos.

### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#mtp-guide)    ⚡ MTP Guide

MTP (Multi Token Prediction)speculative decoding enables models like Qwen3.6 to have **~1.4-2.2x faster generation with** **no change in accuracy**. This enables Qwen3.6 27B and 35B-A3B to have **>1.4x speed-up** over the original baseline which is especially useful for local models.

Unsloth Qwen3.6 MTP GGUFs are no longer in experimental mode, and llama.cpp has merged MTP support. Run directly in [Unsloth Studio’s UI](https://unsloth.ai/docs/models/qwen3.6#unsloth-studio-guide) or via llama.cpp. **Qwen3.6 27B MTP now runs at 160 tokens/s generation and Qwen3.6 35B-A3B at 240 tokens/s on a RTX 6000 GPU.** See [MTP Benchmarks](https://unsloth.ai/docs/models/qwen3.6#mtp-benchmarks).

Unsloth Studio automatically sets the ideal MTP settings optimized for your specific hardware (Mac, CPU, GPU etc.) - you can still change it later.

**MTP uses slightly more VRAM than standard GGUFs**, so plan for ~1 GB additional RAM/VRAM headroom.

[Run in Unsloth Studio](https://unsloth.ai/docs/models/qwen3.6#unsloth-studio-mtp-guide) [Run in llama.cpp](https://unsloth.ai/docs/models/qwen3.6#llama.cpp-mtp-guide) [Run NVFP4](https://unsloth.ai/docs/models/qwen3.6#nvfp4)

[Qwen3.6-27B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF)

[Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF)

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FMYn1g7MDeVRAUa6i2oOk%252Fthroughput%2520mpt.png%3Falt%3Dmedia%26token%3Daff44c0a-3cc3-493e-b6b4-e3279dfb90c1&width=768&dpr=3&quality=100&sign=4d8b316e&sv=2)

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F4CvUpxr0xdxCR1lTGUbS%252Fmtp%2520benchmarks%2520landscape.png%3Falt%3Dmedia%26token%3Dd6579f47-a196-408b-a013-329e09705251&width=768&dpr=3&quality=100&sign=d7cc7fe6&sv=2)

In practice, MTP predicts several future tokens, then the main model verifies those tokens in parallel. This reduces the number of forward passes needed during generation and make output faster. **We found**`--spec-draft-n-max 2` **to work best in most setups.** **However, do not assume**`2` **is optimal, as performance is hardware-dependent. Try values from**`1` **through**`6` **and use whichever is fastest for your system.**

We also [uploaded MTP GGUFs](https://huggingface.co/unsloth/models?search=mtp) for the [**Qwen3.5**](https://unsloth.ai/docs/models/qwen3.5) **model family** including: 0.8B, 2B, 4B, 9B, 27B, 35B-A3B, 122B-A10B and 397B-A17B. Llama.cpp is continually improving MTP performance, so expect it to get faster overtime!

**Table: MTP hardware requirements** (units = total memory: RAM + VRAM, or unified memory)

Qwen3.6

3-bit

4-bit

6-bit

8-bit

BF16

**27B**

16 GB

19 GB

25 GB

31 GB

56 GB

**35B-A3B**

18 GB

24 GB

31 GB

39 GB

71 GB

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#unsloth-studio-mtp-guide)    🦥 Unsloth Studio MTP Guide

Unsloth Studio automatically sets the ideal MTP settings optimized for your specific hardware (Mac, CPU, GPU etc.) - you can still change it later.

1

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#install-unsloth-1)    Install Unsloth

Run in your terminal:

**MacOS, Linux, WSL:**

Copy

```
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows PowerShell:**

Copy

```
irm https://unsloth.ai/install.ps1 | iex
```

2

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#launch-unsloth-1)    Launch Unsloth

**MacOS, Linux, WSL and Windows:**

Copy

```
unsloth studio -H 127.0.0.1 -p 8888
```

Then open `http://127.0.0.1:8888` (or your specific URL) in your browser.

3

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#search-and-download-qwen3.6-mtp)    Search and download Qwen3.6 MTP

On first launch you will need to create a password to secure your account and sign in again later. Then go to the [Studio Chat](https://unsloth.ai/docs/new/studio/chat) tab and search for Qwen3.6 MTP in the search bar and download your desired model and quant.

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F2zQdtdNf0CRBrnTOXBZa%252FScreenshot%25202026-05-16%2520at%25207.10.39%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3Dd1f2e482-5cb7-4e41-97ed-f2905a81f262&width=768&dpr=3&quality=100&sign=c5efb5e3&sv=2)

4

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#run-qwen3.6-mtp)    Run Qwen3.6 MTP

Inference parameters should be auto-set when using Unsloth Studio, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our [Unsloth Studio inference guide](https://unsloth.ai/docs/new/studio/chat). Below, the 2-bit Qwen3.6 MTP GGUF made 10+ tool calls, searched 10 sites and executed Python code:

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FsERzR65n4jc1UtuSHozZ%252Fwedefrwfwe.gif%3Falt%3Dmedia%26token%3De303ed9e-0d90-456d-8d57-874a06803903&width=768&dpr=3&quality=100&sign=711edce8&sv=2)

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#llama.cpp-mtp-guide)    🦙 Llama.cpp MTP Guide

1

Install the latest version of `llama.cpp` on [**GitHub here**](https://github.com/ggml-org/llama.cpp/pull/22673). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

Copy

```
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2

If you want to use `llama.cpp` directly to load models, you can do the below: (:`Q4_K_XL`) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. The model has a maximum of 256K context length.

Follow one of the commands for the specific models:

[27B MTP](https://unsloth.ai/docs/models/qwen3.6#mtp-qwen3.6-27b) [35-A3B MTP](https://unsloth.ai/docs/models/qwen3.6#mtp-qwen3.6-35b-a3b)

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#mtp-qwen3.6-27b)    MTP Qwen3.6-27B:

**Thinking mode:**

Please see Qwen3.6's new [Preserved Thinking](https://unsloth.ai/docs/models/qwen3.6#thinking-enable-disable--preserve-thinking).

General tasks:

Copy

```
export LLAMA_CACHE="unsloth/Qwen3.6-27B-MTP-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --spec-type draft-mtp --spec-draft-n-max 2
```

For precise coding tasks, change: `temperature=0.6`

**Non-thinking mode:**

General tasks:

Copy

```
export LLAMA_CACHE="unsloth/Qwen3.6-27B-MTP-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --presence-penalty 1.5 \
    --min-p 0.00 \
    --spec-type draft-mtp --spec-draft-n-max 2 \
    --chat-template-kwargs '{"enable_thinking":false}'
```

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#mtp-qwen3.6-35b-a3b)    MTP Qwen3.6-35B-A3B:

**Thinking mode:**

Please see Qwen3.6's new [Preserved Thinking](https://unsloth.ai/docs/models/qwen3.6#thinking-enable-disable--preserve-thinking).

General tasks:

Copy

```
export LLAMA_CACHE="unsloth/Qwen3.6-35B-A3B-MTP-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --spec-type draft-mtp --spec-draft-n-max 2
```

For precise coding tasks, change: `temperature=0.6`

**Non-thinking mode:**

General tasks:

Copy

```
export LLAMA_CACHE="unsloth/Qwen3.6-35B-A3B-MTP-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --presence-penalty 1.5 \
    --min-p 0.00 \
    --spec-type draft-mtp --spec-draft-n-max 2 \
    --chat-template-kwargs '{"enable_thinking":false}'
```

3

You can also download the model manually as well via the code below (after installing `pip install huggingface_hub`). You can choose Q4\_K\_M or other quantized versions like `UD-Q4_K_XL` . We recommend using at least 2-bit dynamic quant `UD-Q2_K_XL` to balance size and accuracy. If downloads get stuck, see: [Hugging Face Hub, XET debugging](https://unsloth.ai/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)

Copy

```
hf download unsloth/Qwen3.6-35B-A3B-MTP-GGUF \
    --local-dir unsloth/Qwen3.6-35B-A3B-MTP-GGUF \
    --include "*mmproj-F16*" \
    --include "*UD-Q4_K_XL*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit
```

4

Then run the model in conversation mode:

Copy

```
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.6-35B-A3B-MTP-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3.6-35B-A3B-MTP-GGUF/mmproj-F16.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.00 \
    --top-k 20 \
    --spec-type draft-mtp --spec-draft-n-max 2
```

### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#llama.cpp-guide)    🦙 Llama.cpp Guide

For this guide we will be utilizing Dynamic 4-bit which works great on a 24GB RAM / Mac device for fast inference on [llama.cpp](llama.cpphttps://github.com/ggml-org/llama.cpp). Because the model is only around 72GB at full F16 precision, we won't need to worry much about performance. [See our GGUF collection](https://huggingface.co/collections/unsloth/qwen36).

[27B](https://unsloth.ai/docs/models/qwen3.6#qwen3.6-27b) [35-A3B](https://unsloth.ai/docs/models/qwen3.6#qwen3.6-35b-a3b)

1

Obtain the latest `llama.cpp` **on** [**GitHub here**](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

Copy

```
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2

If you want to use `llama.cpp` directly to load models, you can do the below: (:`Q4_K_XL`) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. The model has a maximum of 256K context length.

Follow one of the commands for the specific models:

[27B](https://unsloth.ai/docs/models/qwen3.6#qwen3.5-27b) [35-A3B](https://unsloth.ai/docs/models/qwen3.6#qwen3.5-35b-a3b)

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#qwen3.6-27b)    Qwen3.6-27B:

**Thinking mode:**

Please see Qwen3.6's new [Preserved Thinking](https://unsloth.ai/docs/models/qwen3.6#thinking-enable-disable--preserve-thinking).

General tasks:

Copy

```
export LLAMA_CACHE="unsloth/Qwen3.6-27B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.6-27B-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

For precise coding tasks, change: `temperature=0.6`

**Non-thinking mode:**

General tasks:

Copy

```
export LLAMA_CACHE="unsloth/Qwen3.6-27B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.6-27B-GGUF:UD-Q4_K_XL \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --presence-penalty 1.5 \
    --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}'
```

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#qwen3.6-35b-a3b)    Qwen3.6-35B-A3B:

**Thinking mode:**

Please see Qwen3.6's new [Preserved Thinking](https://unsloth.ai/docs/models/qwen3.6#thinking-enable-disable--preserve-thinking).

General tasks:

Copy

```
export LLAMA_CACHE="unsloth/Qwen3.6-35B-A3B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

For precise coding tasks, change: `temperature=0.6`

**Non-thinking mode:**

General tasks:

Copy

```
export LLAMA_CACHE="unsloth/Qwen3.6-35B-A3B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --presence-penalty 1.5 \
    --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}'
```

3

You can also download the model manually as well via the code below (after installing `pip install huggingface_hub`). You can choose Q4\_K\_M or other quantized versions like `UD-Q4_K_XL` . We recommend using at least 2-bit dynamic quant `UD-Q2_K_XL` to balance size and accuracy. If downloads get stuck, see: [Hugging Face Hub, XET debugging](https://unsloth.ai/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)

Copy

```
hf download unsloth/Qwen3.6-35B-A3B-GGUF \
    --local-dir unsloth/Qwen3.6-35B-A3B-GGUF \
    --include "*mmproj-F16*" \
    --include "*UD-Q4_K_XL*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit
```

4

Then run the model in conversation mode:

Copy

```
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3.6-35B-A3B-GGUF/mmproj-F16.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.00 \
    --top-k 20
```

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#llama-server-and-openai-completion-library)    Llama-server & OpenAI completion library

To deploy Qwen3.6 for production, we use `llama-server` In a new terminal say via tmux, deploy the model via:

Copy

```
./llama.cpp/llama-server \
--model unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3.6-35B-A3B-GGUF/mmproj-F16.gguf \
    --alias "unsloth/Qwen3.6-35B-A3B" \
    --temp 0.6 \
    --top-p 0.95 \
    --ctx-size 16384 \
    --top-k 20 \
    --min-p 0.00 \
    --port 8001
```

Then in a new terminal, after doing `pip install openai`, do:

Copy

```
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/Qwen3.6-35B-A3B",
    messages = [{"role": "user", "content": "Create a Snake game."},],
)
print(completion.choices[0].message.content)
```

### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#mlx-dynamic-quants)    🍎 MLX Dynamic Quants

We also uploaded dynamic Qwen3.6 4bit and 8bit quants for MacOS devices! Our MLX quant algorithm is still evolving, and we’re actively refining it wherever improvements can be made.

You can run all MLX models in [Unsloth Studio](https://unsloth.ai/docs/models/qwen3.6#unsloth-studio-guide)!

**Qwen3.6-27B MLX:**

[3-bit](https://huggingface.co/unsloth/Qwen3.6-27B-UD-MLX-3bit)

[4-bit](https://huggingface.co/unsloth/Qwen3.6-27B-UD-MLX-4bit)

[MXFP4](https://huggingface.co/unsloth/Qwen3.6-27B-UD-MLX-MXFP4)

[NVFP4](https://huggingface.co/unsloth/Qwen3.6-27B-UD-MLX-NVFP4)

[6-bit](https://huggingface.co/unsloth/Qwen3.6-27B-UD-MLX-6bit)

[8-bit](https://huggingface.co/unsloth/Qwen3.6-27B-MLX-8bit)

**Qwen3.6-35B-A3B MLX:**

[3-bit](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-UD-MLX-3bit)

[4-bit](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit)

[8-bit](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MLX-8bit)

To try them out use:

Copy

```
curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/scripts/install_qwen3_6_mlx.sh | sh
source ~/.unsloth/unsloth_qwen3_6_mlx/bin/activate
python -m mlx_vlm.chat --model unsloth/Qwen3.6-27B-UD-MLX-4bit
```

See below for Qwen3.6-27B KL Divergence (KLD) and Perplexity (PPL) scores (lower is better):

Model

Mean KLD

Median KLD

PPL

P90 KLD

P99.9 KLD

Size

[8-bit](https://huggingface.co/unsloth/Qwen3.6-27B-MLX-8bit)

0.0028

0.0003

4.812

0.0019

0.192

34.7 GB

[6-bit](https://huggingface.co/unsloth/Qwen3.6-27B-UD-MLX-6bit)

0.0037

0.0007

4.809

0.0032

0.343

30.5 GB

[4-bit](https://huggingface.co/unsloth/Qwen3.6-27B-UD-MLX-4bit)

0.0227

0.0053

4.821

0.0293

2.339

26.2 GB

[NVFP4](https://huggingface.co/unsloth/Qwen3.6-27B-UD-MLX-NVFP4)

0.0325

0.0087

4.843

0.0466

3.693

26.2 GB

[MXFP4](https://huggingface.co/unsloth/Qwen3.6-27B-UD-MLX-MXFP4)

0.0479

0.0153

4.902

0.0769

4.035

25.6 GB

[3-bit](https://huggingface.co/unsloth/Qwen3.6-27B-UD-MLX-3bit)

0.0734

0.0223

4.976

0.1261

5.529

24.1 GB

### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#nvfp4)    ⚡️NVFP4

You can now run our NVFP4 quants with MTP tensors directly integrated inside the NVFP4 quant. Both [vLLM](https://unsloth.ai/docs/basics/inference-and-deployment/vllm-guide) and [SGLang](https://unsloth.ai/docs/basics/inference-and-deployment/sglang-guide) work for this. We tried `vllm==0.22.0` and `sglang==0.5.9` (you may need to use SGLang main).

- Qwen3.6-35B-A3B NVFP4: [huggingface.co/unsloth/Qwen3.6-35B-A3B-NVFP4](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-NVFP4)

- Qwen3.6-27B NVFP4: [huggingface.co/unsloth/Qwen3.6-27B-NVFP4](https://huggingface.co/unsloth/Qwen3.6-27B-NVFP4)


#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#vllm)    **vLLM:**

Copy

```
vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4 --trust-remote-code --dtype bfloat16 --moe-backend marlin \
     --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}'
```

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#sglang)    **SGLang:**

Copy

```
python -m sglang.launch_server --model-path unsloth/Qwen3.6-27B-NVFP4 --speculative-algo NEXTN \
     --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
```

### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#thinking-enable-disable--preserve-thinking)    💡 Thinking: Enable/Disable + Preserve Thinking

Qwen3.6 also has **Preserve Thinking** which leaves the thinking trace from the previous conversation. This increases the number of tokens you use, but could increase accuracy in continued conversations. Unsloth Studio has 'Think' and Preserved Thinking toggles for Qwen3.6:

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FKoonE97b0H3RioMwVohd%252Fstudio%2520new%2520ui.gif%3Falt%3Dmedia%26token%3De37f2839-914e-48b8-8c81-2dac3ade9408&width=768&dpr=3&quality=100&sign=2c00972a&sv=2)

Unsloth Studio has Think toggle by default and a new [Preserved Thinking](https://unsloth.ai/docs/models/qwen3.6#preserved-thinking) toggle

To enable **preserve thinking** in llama.cpp use (change to 'true' or 'false') '`preserve_thinking`' instead of '`enable_thinking`' or '`disable_thinking`'.

Copy

```
--chat-template-kwargs '{"preserve_thinking":true}'
```

For normal thinking, you can enable / disable thinking in llama.cpp by following the below commands. Use '`true`' and '`false`' interchangeably.

llama-server OS:

Enable Thinking

Disable Thinking

Linux, MacOS, WSL:

Copy

```
--chat-template-kwargs '{"enable_thinking":true}'
```

Copy

```
--chat-template-kwargs '{"enable_thinking":false}'
```

Windows / Powershell:

Copy

```
--chat-template-kwargs "{\"enable_thinking\":true}"
```

Copy

```
--chat-template-kwargs "{\"enable_thinking\":false}"
```

As an example for Qwen3.6-35B-A3B to enable preserve thinking (default is enabled):

Copy

```
./llama.cpp/llama-server \
    --model unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-BF16.gguf \
    --alias "unsloth/Qwen3.6-35B-A3B-GGUF" \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --port 8001 \
    --chat-template-kwargs '{"preserve_thinking":true}'
```

And then in Python:

Copy

```
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/Qwen3.6-35B-A3B-GGUF",
    messages = [{"role": "user", "content": "What is 2+2?"},],
)
print(completion.choices[0].message.content)
print(completion.choices[0].message.reasoning_content)
```

### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#claude-codex)    👨‍💻 OpenAI Codex & Claude Code

To run the model via local coding agentic workloads, you can [follow our guide](https://unsloth.ai/docs/models/qwen3.6#claude-codex). Use the `llama-server` we just set up just then, and set the model name to the exact id it reports at `GET /v1/models` (the `--alias` value above, e.g. `unsloth/Qwen3.6-35B-A3B-GGUF`). Follow the correct Qwen3.6 parameters and usage instructions.

[Claude Code](https://unsloth.ai/docs/basics/claude-code)

[OpenAI Codex](https://unsloth.ai/docs/basics/codex)

After following the instructions for Claude Code for example you will see:

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fup2DMSMPjNR8BM9pgR0v%252Fimage.png%3Falt%3Dmedia%26token%3D152e9ee0-2491-4379-af18-8fca0789b19d&width=768&dpr=3&quality=100&sign=580b633c&sv=2)

We can then ask say `Create a Python game for Chess` :

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F9TfMAoKSdMpb8OHKNnHH%252Fimage.png%3Falt%3Dmedia%26token%3D771df3aa-91ab-4c1e-8676-1830058001ca&width=768&dpr=3&quality=100&sign=efa8210a&sv=2)

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FWP3lI5mQW2EHB79qqgDz%252Fimage.png%3Falt%3Dmedia%26token%3D55cf3189-e100-419c-a615-024b45948284&width=768&dpr=3&quality=100&sign=96b8fc09&sv=2)

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fn8DZddDODQZGCP8giKYY%252Fimage.png%3Falt%3Dmedia%26token%3D996c8cb9-d199-4045-90f0-408690e02667&width=768&dpr=3&quality=100&sign=59b9ff10&sv=2)

## [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#benchmarks)    📊 Benchmarks

### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#unsloth-gguf-benchmarks)    Unsloth GGUF Benchmarks

We conducted Mean KL Divergence benchmarks for Qwen3.6-35-A3B GGUFs across providers to help you pick the best quant.

- KL Divergence puts nearly all Unsloth GGUFs on the SOTA Pareto frontier

- KLD shows how well a quantized model matches the original BF16 output distribution, indicating retained accuracy.

- This makes Unsloth the top-performing in 21 of 22 sizes

- Only Q6\_K was updated for more Dynamic layers and we introduced a new `UD-IQ4_NL_XL` quant


![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FHq98A18pHA2ePwlInrFG%252Fqwen36_mean_q6k_corrected_arrow_pareto_fixed.png%3Falt%3Dmedia%26token%3Da5190c8a-4d04-4d4d-be94-dd15214e6687&width=768&dpr=3&quality=100&sign=d0938e2f&sv=2)

35B-A3B - KLD benchmarks (lower is better)

### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#mtp-benchmarks)    MTP Benchmarks

We benchmarked the new quants we made for 27B and 35B MoE. In general, dense models are much more accelerated with MTP (1.4-2x) vs MoE models (1.15-1.25x).

With this, Qwen3.6 27B can now do 140 tokens / s generation with UD-Q2\_K\_XL and Qwen3.6 35B-A3B 220 tokens / s generation! Some of the throughput numbers are noisy, so don't infer some quants are slower than others.

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FfXF043cvTRVLrH2MLqHg%252FQwen36_mtp_spec_throughput_side_by_side.png%3Falt%3Dmedia%26token%3Dd2275610-fe4e-4af0-973b-0881a044f775&width=768&dpr=3&quality=100&sign=9124743f&sv=2)

In terms of average speedup, we see a 1.4x for dense models at draft tokens = 2 and for the MoE around 1.15 to 1.2x.

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F7arbIuA74e9Tbv7ISRcJ%252FQwen36_mtp_spec_speedup_side_by_side_black_solid_average.png%3Falt%3Dmedia%26token%3D92bd8705-3842-4aef-90e7-0e61b9d50fba&width=768&dpr=3&quality=100&sign=a0b31aa0&sv=2)

We do not recommend more than 2 draft tokens because the acceptance rate drops precipitously from 83% to 50% with 4 draft tokens, and the forward passes for MTP become less beneficial.

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F0n6KhVbrWqhwBbSUj0JJ%252Foutput%2834%29.png%3Falt%3Dmedia%26token%3Daa6adf45-f3d6-4541-bab0-8f9697c0e66c&width=768&dpr=3&quality=100&sign=dfd965b2&sv=2)

### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#official-qwen-benchmarks)    Official Qwen Benchmarks

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#qwen3.6-27b-1)    Qwen3.6-27B

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FlvrSCq6GKFGADGSOTE9W%252Fqwen3.6_27b_score.png%3Falt%3Dmedia%26token%3D6f1abf24-6a15-4988-a305-acd5638aaf0d&width=768&dpr=3&quality=100&sign=a848cf34&sv=2)

#### [Direct link to heading](https://unsloth.ai/docs/models/qwen3.6\#qwen3.6-35b-a3b-1)    Qwen3.6-35B-A3B

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F25aKI2tJR2PNfGfwnbZi%252Fqwen3.6_35b_a3b_score%282%29.png%3Falt%3Dmedia%26token%3Df296d01d-311d-413e-8c62-122728e33008&width=768&dpr=3&quality=100&sign=ecfe4c2c&sv=2)

[PreviousFine-tune Gemma 4](https://unsloth.ai/docs/models/gemma-4/train) [NextKimi K2.7 Code](https://unsloth.ai/docs/models/kimi-k2.7-code)

Last updated 11 days ago

Was this helpful?

This site uses cookies to deliver its service and to analyze traffic. By browsing this site, you accept the [privacy policy](https://policies.gitbook.com/privacy/cookies).

AcceptReject