# Ornith-1.0-9B MTP — GGUF (llama.cpp speculative decoding)

GGUF builds of [`deepreinforce-ai/Ornith-1.0-9B`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B) **with the KL-distilled MTP draft head** from
[`protoLabsAI/Ornith-1.0-9B-MTP`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP) baked
in — so llama.cpp does **lossless multi-token (self-)speculative decoding** out of the box, no
separate draft model required.

**~1.4–1.7× single-stream decode speedup on a single RTX A6000**, distribution-lossless. The
head's per-token acceptance on llama.cpp matches the vLLM reference ( **0.766** here vs 0.762).

> Just want the base model with no MTP? Use `deepreinforce-ai/Ornith-1.0-9B-GGUF`. These files
> add the `nextn` head on top of the same trunk.

## NEW: NVFP4 build — 306 tok/s on Blackwell, the fastest rung in this repo

`Ornith-1.0-9B-MTP-NVFP4.gguf` (6.6 GB) — calibrated NVFP4 (`GGML_TYPE_NVFP4`) on all
attention/MLP GEMMs, Q8\_0 on the DeltaNet trunk/embeddings, MTP `nextn` head included.
Converted directly from our gate-verified
[vLLM NVFP4 quant](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-NVFP4) — same calibrated
scales; full quality/coherence receipts on that card.

Measured on both hardware classes, `-n 200` greedy, MTP = `--spec-type draft-mtp`.
Blackwell MTP numbers are means over 6 diverse prompts (prose/code/creative/factual/
runbook/technical); ranges shown. Ampere numbers are single-run indicative.

```
                   Ampere A6000       Blackwell (sm120)
file       size    no-MTP   +MTP      no-MTP   +MTP
---------  ------  ------   ------    ------   ----------------
Q4_K_M     5.8 GB  104.6    153.4     205.1    239  (216-252)
NVFP4-MTP  6.6 GB   70.7     84.8     201.5    306  (287-330)
```

**Read this honestly:** on Ampere and older, Q4\_K\_M is smaller AND faster — use it. On
Blackwell (RTX 50xx / PRO 6000), NVFP4+MTP is the fastest rung in this repo by ~28%
(worst NVFP4 prompt beats best Q4\_K\_M prompt). **Why — measured, acceptance-controlled:**
draft acceptance is near-equal on both files (0.52 vs 0.49, same prompt/box), so the
differential is per-step verify cost: a 2-token MTP verify step costs ~0% extra on
NVFP4's tensor-core GEMMs vs ~+28% on the K-quant dequant path. MTP's speedup is
effectively multiplicative with NVFP4 and only partial with K-quants — a differential
we haven't seen measured elsewhere (spec-decode x FP8 compounding is documented by
TensorRT-LLM; the FP4-vs-K-quant verify-cost split appears to be new data).
Workload note: code prompts run hottest (330), creative prose lowest (287) — MTP
acceptance tracks predictability. Requires a llama.cpp build with NVFP4 (type 40) and
MTP support (both merged spring 2026).

## Files

| File | Form | Size | Use |
| --- | --- | --: | --- |
| `Ornith-1.0-9B-MTP-Q8_0.gguf` | bundled (trunk + head) | 9.8 GB | highest quality / biggest relative speedup |
| `Ornith-1.0-9B-MTP-Q6_K.gguf` | bundled | 7.6 GB | near-lossless quant |
| `Ornith-1.0-9B-MTP-Q5_K_M.gguf` | bundled | 6.6 GB | balanced |
| `Ornith-1.0-9B-MTP-Q4_K_M.gguf` | bundled | 5.8 GB | fastest k-quant |
| `Ornith-1.0-9B-MTP-IQ4_XS.gguf` | bundled (imatrix) | 5.5 GB | low VRAM, near-Q4 quality |
| `Ornith-1.0-9B-MTP-IQ3_M.gguf` | bundled (imatrix) | 4.7 GB | lower VRAM |
| `Ornith-1.0-9B-MTP-IQ2_M.gguf` | bundled (imatrix) | 3.9 GB | **very low VRAM** (~5 GB to serve) |
| `Ornith-1.0-9B-MTP-BF16.gguf` | bundled (full precision) | 18.4 GB | the master; re-quantize from this |
| `mtp-Ornith-1.0-9B-MTP-Q8_0.gguf` | standalone draft head | 2.4 GB | attach to a base GGUF via `--model-draft` |

The **IQ** quants are i-quants built with an importance matrix (calibrated on the trunk) for
quality at low bit-rates, with the **MTP `nextn` head pinned to Q8\_0** so speculative-decode
acceptance holds even on the 2-bit trunk (verified ~0.81–0.84 accept on IQ2\_M–IQ4\_XS, on par
with the k-quants). Serve them exactly like the k-quants (`--spec-type draft-mtp`).

Requires **llama.cpp ≥ b9616** (Qwen3.5 `qwen35` arch + `--spec-type draft-mtp`).

## Run

**Bundled (recommended)** — the head travels in the file:

```bash
llama-server --model Ornith-1.0-9B-MTP-Q4_K_M.gguf \
  --n-gpu-layers 99 --ctx-size 8192 --flash-attn on --jinja \
  --spec-type draft-mtp --spec-draft-n-max 3
```

**Standalone draft** — pair the small head with any base Ornith-9B GGUF:

```bash
llama-server --model ornith-1.0-9b-Q4_K_M.gguf \
  --model-draft mtp-Ornith-1.0-9B-MTP-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --n-gpu-layers 99 --ctx-size 8192 --flash-attn on --jinja
```

`--spec-draft-n-max` is the draft depth: **2** maximizes acceptance, **3** maximizes
throughput, 4 starts to regress. Tune per workload.

## Benchmarks (RTX A6000, ctx 8192, flash-attn, greedy; 6-prompt code+general mix)

### n-max sweep (Q8\_0)

| config | decode tok/s | acceptance | speedup |
| --- | --: | --: | --: |
| base (no MTP) | 71.0 | — | 1.00× |
| MTP n-max 2 | 118.3 | **0.766** | 1.67× |
| **MTP n-max 3** | **122.6** | 0.651 | **1.73×** |
| MTP n-max 4 | 120.8 | 0.565 | 1.70× |

### Across quants (MTP n-max 3)

| quant | base tok/s | MTP tok/s | speedup | acceptance |
| --- | --: | --: | --: | --: |
| Q4\_K\_M | 105.4 | **145.3** | 1.38× | 0.659 |
| Q8\_0 | 71.0 | 122.6 | 1.73× | 0.651 |

Acceptance is **quant-stable** (~0.65 @ n-max 3 even with the Q4 head). Q4\_K\_M is fastest in
absolute terms; the _relative_ MTP gain grows with precision (Q8's slow bandwidth-bound
baseline has more to gain from the parallel verify).

## "Lossless" — read this

MTP speculative decoding is **distribution-lossless**: every drafted token is verified against
the target, so the output distribution is unchanged. It is **not bitwise-identical** to plain
decode at greedy/temp 0 — the batched verification path computes target logits in a different
floating-point reduction order than sequential decoding, which can flip a greedy argmax and
fork the text. Both outputs are equally valid and equal quality; this is expected llama.cpp
behavior, not a defect of these weights.

## How these were built

```bash
# 1. graft the mtp.* head into the base trunk (15 tensors, 1 nextn layer)
python graft.py --donor protoLabsAI/Ornith-1.0-9B-MTP \
                --target deepreinforce-ai/Ornith-1.0-9B --out ./ornith-9b-mtp-kl
# 2. convert (the converter remaps mtp.* -> blk.<32>.nextn.* automatically)
python convert_hf_to_gguf.py ./ornith-9b-mtp-kl --outfile out/...-BF16.gguf --outtype bf16
python convert_hf_to_gguf.py ./ornith-9b-mtp-kl --outfile out/ --outtype q8_0 --mtp   # standalone draft
# 3. quantize
llama-quantize out/...-BF16.gguf out/...-Q4_K_M.gguf Q4_K_M
```

The `graft.py` recipe and the KL-distillation details live in the head repo
[`protoLabsAI/Ornith-1.0-9B-MTP`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP).

### Common error: `wrong number of tensors expected 442 got 427`

(or `got 426` for the smaller quants — the gap is the **15 `mtp.*` head tensors**.)

This happens if you run `convert_hf_to_gguf.py` directly on the **base**`deepreinforce-ai/Ornith-1.0-9B` without grafting the head first. The base keeps
`mtp_num_hidden_layers: 1` in its `config.json` (`text_config`) but **ships none of the `mtp.*`**
**weights** — so the converter writes `block_count = 33` / `nextn_predict_layers = 1` into the
GGUF metadata (declaring the `blk.32` MTP layer) while leaving those 15 tensors empty. llama.cpp
then expects 442 tensors and finds 427 → load fails.

**Fix:** graft the head into the trunk _before_ converting (step 1 above), then convert with no
`--mtp` flag. Note that only 4 of the 15 head tensors are named `blk.32.nextn.*` (`eh_proj`,
`enorm`, `hnorm`, `shared_head_norm`); the other 11 land as ordinary `blk.32.*` layer tensors
(`attn_*`, `ffn_*`, the norms) — so grepping for `nextn` shows only 4, but the head is complete.

Don't want to graft? You don't have to build the bundled file at all — run the base GGUF with
`--model-draft mtp-Ornith-1.0-9B-MTP-Q8_0.gguf --spec-type draft-mtp`. Functionally identical.

## Provenance & license

- **Base:**`deepreinforce-ai/Ornith-1.0-9B` (MIT) — a Qwen3.5-9B hybrid (linear-attention +
full-attention) fine-tune.
- **MTP head:**`protoLabsAI/Ornith-1.0-9B-MTP` (MIT) — KL-distilled against Ornith's own
hidden states.
- These GGUFs are a derivative of both; **MIT**. Built by [protoLabs.studio](https://protolabs.studio/).

## Want a different size or format?

Open a Community discussion — requests usually ship within 48h. That's how most of the
quants in this repo got here. All benchmark rows behind these numbers:
[`protoLabsAI/lab-benchmarks`](https://huggingface.co/datasets/protoLabsAI/lab-benchmarks) ·
charts at [protolabs.studio/lab](https://protolabs.studio/lab).

## Which file do I download?

| your GPU VRAM | file | notes |
| --- | --- | --- |
| 24 GB+ (RTX 50xx/Blackwell) | `Ornith-1.0-9B-MTP-NVFP4.gguf` | fastest rung, 306 tok/s w/ MTP |
| 12–24 GB | `Ornith-1.0-9B-MTP-Q8_0.gguf` or `Q6_K` | reference quality |
| 8–12 GB | `Q5_K_M` / `Q4_K_M` / `IQ4_XS` | best size/quality trade |
| 4–6 GB | `IQ2_M` (3.9 GB) | partial offload (`-ngl 20`, small `-c`); expect modest speed. A 4B-class model may serve you better |
| — | `mtp-head/mtp-Ornith-1.0-9B-head-Q8_0.gguf` | ⚠️ **NOT a model** — standalone MTP draft head for `--model-draft`; loading it directly will crash |

## One-command usage

```bash
llama-server -hf protoLabsAI/Ornith-1.0-9B-MTP-GGUF:NVFP4 --spec-type draft-mtp -ngl 99   # Blackwell: fastest
llama-server -hf protoLabsAI/Ornith-1.0-9B-MTP-GGUF:Q4_K_M --spec-type draft-mtp -ngl 99  # everything else
```

> 2026-07-03: files renamed to the standard `<Model>-<QUANT>.gguf` convention — older llama.cpp
> `-hf` invocations with the previous `ornith-9b-mtp-kl-*` filenames should switch to tags.

Downloads last month16,824

GGUF

Model size

9B params

Architecture

qwen35

Chat template

Hardware compatibility

[Log In](https://huggingface.co/login) to add your hardware

2-bit

MTPIQ2\_M

3.87 GB

3-bit

MTPIQ3\_M

4.67 GB

4-bit

MTPIQ4\_XS

5.45 GBMTPNVFP4

6.63 GBMTPQ4\_K\_M

5.78 GB

5-bit

MTPQ5\_K\_M

6.64 GB

6-bit

MTPQ6\_K

7.56 GB

8-bit

MTPQ8\_0

9.79 GBMTPQ8\_0

2.43 GB

16-bit

MTPBF16

18.4 GB

Inference Providers [NEW](https://huggingface.co/docs/inference-providers)

[Text Generation](https://huggingface.co/tasks/text-generation "Learn more about text-generation")

This model isn't deployed by any Inference Provider. [🙋Ask for provider support](https://huggingface.co/spaces/huggingface/InferenceSupport/discussions/new?title=protoLabsAI/Ornith-1.0-9B-MTP-GGUF&description=React%20to%20this%20comment%20with%20an%20emoji%20to%20vote%20for%20%5BprotoLabsAI%2FOrnith-1.0-9B-MTP-GGUF%5D(%2FprotoLabsAI%2FOrnith-1.0-9B-MTP-GGUF)%20to%20be%20supported%20by%20Inference%20Providers.%0A%0A(optional)%20Which%20providers%20are%20you%20interested%20in%3F%20(Novita%2C%20Hyperbolic%2C%20Together%E2%80%A6)%0A)

## Model tree for protoLabsAI/Ornith-1.0-9B-MTP-GGUF

Base model

[deepreinforce-ai/Ornith-1.0-9B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B)

Quantized

( [59](https://huggingface.co/models?other=base_model:quantized:deepreinforce-ai/Ornith-1.0-9B))

this model