# Ornith-1.0-35B — GGUF (llama.cpp, single-GPU `tp=1`)

Single-GPU llama.cpp GGUF package for [`deepreinforce-ai/Ornith-1.0-35B`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B).
The supported serving policy is `tp=1` — one model copy per GPU. Multi-GPU
tensor-parallel serving is intentionally out of scope for this version.

This release ships six body quants (Q3\_K\_M → Q8\_0) plus an **integrated**
**IQ4\_XS-MTP graft** that adds a native multi-token-prediction (MTP) draft head for
low-concurrency speculative decode. Quant quality is measured against the upstream
BF16 GGUF with a native llama.cpp next-token top-64 KL-divergence probe over 32
coding prompts.

## TL;DR — pick an artifact

| Use case | Recommended artifact | Key numbers |
| --- | --- | --- |
| Default serving speed | `ornith-1.0-35b-Q4_K_M.gguf` | 19.71 GiB on disk, 21.31 GiB loaded VRAM, 243.3 tok/s c1, 655.6 tok/s c16 |
| Lowest memory | `ornith-1.0-35b-Q3_K_M.gguf` | 15.61 GiB on disk, 17.27 GiB loaded VRAM, 240.5 tok/s c1, 493.0 tok/s c16 |
| Middle footprint | `ornith-1.0-35b-IQ4_XS.gguf` | 17.64 GiB on disk, 19.34 GiB loaded VRAM, 0.1426 mean top-64 KLD nats |
| Highest fidelity / footprint | `ornith-1.0-35b-Q6_K.gguf` | 26.56 GiB on disk, 28.03 GiB loaded VRAM, 0.0165 mean top-64 KLD nats, 32/32 top-1 |
| Native low-concurrency MTP | `ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf` | 19.6 GB decimal, **0.0731 mean top-64 KLD (29/32)**, c1/128 adaptive MTP 319.53 tok/s, acceptance `(0.953, 0.865)` |

> **MTP graft note:** active MTP is byte-for-byte identical to target-only at the
> next-token (API-visible) level, and gives ~1.3× single-stream decode. It is
> **not** bit-exact to target-only over long deterministic sequences — see
> [MTP](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1#multi-token-prediction-mtp). Use target-only for strict reproduction,
> MTP for throughput on open-ended work.

## Quantization & fidelity

Fidelity is `KL(P_bf16 || P_candidate)` over next-token top-64 distributions,
32 coding prompts, `n_predict=1, temperature=-1, n_probs=64`, keyed by token ID.
Lower KLD and higher greedy top-1 agreement are better.

### Primary fidelity table

| Artifact | Mean KLD (nats) | Greedy top-1 † |
| --- | --: | --: |
| `Q8_0` | 0.0108 | 31/32 (96.9%) |
| `Q6_K` | 0.0165 | 32/32 (100.0%) |
| `Q5_K_M` | 0.0354 | 30/32 (93.8%) |
| **`IQ4_XS-MTP` (graft)** | **0.0731382442** | **29/32 (90.6%)** |
| `Q4_K_M` | 0.0864 | 29/32 (90.6%) |
| `IQ4_XS` | 0.1426 | 27/32 (84.4%) |
| `Q3_K_M` | 0.3620 | 27/32 (84.4%) |

[![Next-token fidelity ladder: mean top-64 KLD vs BF16 for the six body quants plus the IQ4_XS-MTP graft, lower is better.](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/resolve/main/assets/02_fidelity_ladder.png)](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/assets/02_fidelity_ladder.png)

_Fidelity ladder — the integrated IQ4\_XS-MTP graft (purple) lands between Q5\_K\_M_
_and Q4\_K\_M, i.e. better next-token fidelity than its Q4\_K\_M neighbor._

> **† Top-1 vs mean KLD are different axes.** _Top-1_ counts how often the single
> most-likely (argmax) token matches BF16's, over the 32 prompts; _mean KLD_
> measures divergence across the full top-64 distribution. Two quants can share a
> top-1 count yet differ a lot in KLD — e.g. `Q3_K_M` and `IQ4_XS` are both
> 27/32 on top-1, but `Q3_K_M`'s distribution diverges ~2.5× more (0.3620 vs
> 0.1426 nats). Neither value is a typo.

- The **integrated IQ4\_XS-MTP graft is 48.70% lower mean KLD than the plain**
**IQ4\_XS body** (0.0731382442 vs 0.1425748206), and improves greedy top-1 from
27/32 to 29/32.
- **Active `draft-mtp` ≡ target-only at the next-token level:** their top-64
next-token distributions are identical (KLD 0.0, 32/32) for the one-token
API-visible probe.

### Body-quant distribution detail (secondary)

Full per-quant distribution from the corrected top-64 probe (same 32 prompts).
`Mean / P50 / P95 / Max` are explicit nats statistics of the per-prompt top-64
KLD; in this run the BF16 top-64 distribution captured mean probability mass
0.999965, so the top-64 approximation captured essentially all next-token mass.

| Quant | GGUF GiB | KLD-probe VRAM GiB | Mean nats | P50 nats | P95 nats | Max nats | Top-1 † |
| --- | --: | --: | --: | --: | --: | --: | --: |
| Q3\_K\_M | 15.61 | 16.32 | 0.3620 | 0.2548 | 1.1077 | 1.3730 | 27/32 (84.4%) |
| IQ4\_XS | 17.64 | 19.34 | 0.1426 | 0.0868 | 0.3195 | 0.6586 | 27/32 (84.4%) |
| Q4\_K\_M | 19.71 | 20.35 | 0.0864 | 0.0379 | 0.2877 | 0.4503 | 29/32 (90.6%) |
| Q5\_K\_M | 23.03 | 23.61 | 0.0354 | 0.0235 | 0.0943 | 0.2497 | 30/32 (93.8%) |
| Q6\_K | 26.56 | 27.07 | 0.0165 | 0.0092 | 0.0513 | 0.0586 | 32/32 (100.0%) |
| Q8\_0 | 34.37 | 34.77 | 0.0108 | 0.0052 | 0.0440 | 0.0590 | 31/32 (96.9%) |

The IQ4\_XS-MTP graft uses the same probe on a different breakpoint context; its
distribution detail (`P50 0.0492811974 / P95 0.1593743642 / Max 0.3348258511`) is
in the [MTP](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1#multi-token-prediction-mtp) section. Source:
[benchmarks/kld-quant-vs-bf16-top64.md](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/benchmarks/kld-quant-vs-bf16-top64.md).

## Serving performance

Aggregate decode throughput and p95 time-to-first-token across concurrency,
llama.cpp `tp=1`, from `scripts/bench_openai.py --stream --max-tokens 256` (0
failed requests per row). Q4\_K\_M is the speed pick; Q5\_K\_M nearly ties it at c16.

| Quant | c1 tok/s | c1 p95 TTFT ms | c4 tok/s | c4 p95 TTFT ms | c8 tok/s | c8 p95 TTFT ms | c16 tok/s | c16 p95 TTFT ms |
| --- | --: | --: | --: | --: | --: | --: | --: | --: |
| Q3\_K\_M | 240.5 | 77.9 | 422.0 | 170.7 | 464.4 | 344.9 | 493.0 | 493.7 |
| IQ4\_XS | 234.1 | 75.1 | 297.7 | 159.7 | 411.5 | 330.2 | 476.0 | 541.5 |
| Q4\_K\_M | 243.3 | 76.3 | 458.3 | 192.3 | 615.0 | 361.8 | 655.6 | 650.0 |
| Q5\_K\_M | 236.7 | 75.1 | 311.0 | 198.8 | 439.0 | 383.6 | 638.6 | 620.4 |
| Q6\_K | 225.9 | 76.8 | 295.8 | 194.2 | 409.6 | 394.8 | 603.3 | 657.4 |
| Q8\_0 | 208.5 | 76.9 | 281.5 | 190.8 | 405.8 | 389.1 | 601.4 | 725.8 |

[![Aggregate decode throughput vs concurrency (c1–c16) for the six quants; Q4_K_M leads at c16.](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/resolve/main/assets/03_throughput_tps.png)](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/assets/03_throughput_tps.png)

[![p95 time-to-first-token vs concurrency (c1–c16) for the six quants; lower is better.](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/resolve/main/assets/04_ttft_p95.png)](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/assets/04_ttft_p95.png)

> **Profile note:** the table above is a **short-context** profile — `CTX_SIZE=8192 PARALLEL=16` exposes `n_ctx = 512`/slot, with ~23-token prompts and 256-token
> generations. Source: [benchmarks/llamacpp-quant-benchmarks.md](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/benchmarks/llamacpp-quant-benchmarks.md).
> For how prefill latency scales with prompt length, see long-context TTFT below.

### Long-context TTFT (single stream)

p95 time-to-first-token vs prompt/context length, single GPU `tp=1`, single stream
(`CTX_SIZE=131072 PARALLEL=1`, `n_ctx/slot=131072`, exact prompt token counts, 140
rows/quant, 0 failures).

| Context tokens | Q4\_K\_M p50/p95 ms | Q4\_K\_M tok/s | IQ4\_XS p50/p95 ms | IQ4\_XS tok/s | MTP-graft p50/p95 ms | MTP-graft tok/s |
| --: | --: | --: | --: | --: | --: | --: |
| 512 | 91.8 / 94.1 | 188.3 | 87.1 / 88.7 | 177.4 | 86.0 / 87.8 | 184.0 |
| 1024 | 169.6 / 172.2 | 145.8 | 159.9 / 161.6 | 139.9 | 157.6 / 159.0 | 155.9 |
| 2048 | 341.8 / 346.1 | 97.9 | 316.1 / 318.7 | 112.0 | 307.6 / 308.9 | 113.9 |
| 4096 | 699.2 / 702.2 | 67.2 | 647.0 / 653.4 | 70.7 | 625.7 / 631.3 | 69.0 |
| 8192 | 1447.5 / 1458.0 | 33.7 | 1342.9 / 1348.5 | 39.8 | 1297.4 / 1304.0 | 40.9 |
| 16384 | 3009.7 / 3030.4 | 19.5 | 2793.2 / 2804.0 | 20.9 | 2709.8 / 2737.3 | 21.5 |
| 32768 | 6302.4 / 6313.0 | 9.7 | 5829.9 / 5853.1 | 10.5 | 5673.5 / 5696.6 | 10.7 |

[![Long-context p95 TTFT vs prompt length (512–32768 tokens) for Q4_K_M, IQ4_XS, and the MTP graft; single stream, tp=1.](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/resolve/main/assets/07_longctx_ttft.png)](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/assets/07_longctx_ttft.png)

p95 TTFT rises with prompt length (prefill cost) to ~6.3 s at 32k tokens for
Q4\_K\_M; the IQ4\_XS body and IQ4\_XS-MTP graft prefill slightly faster at every
length. Decode throughput falls from ~180–190 tok/s at 512 tokens to ~10 tok/s at
32k as the KV cache grows. Source: [benchmarks/llamacpp-longctx-ttft.md](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/benchmarks/llamacpp-longctx-ttft.md).

## Multi-token prediction (MTP)

The integrated `IQ4_XS-MTP-graft-headQ6` artifact carries a native MTP draft head
on the IQ4\_XS body. The release recommendation is **adaptive**: use MTP for
low-concurrency / single-user requests and keep `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1`
so saturated batches fall back to target-only throughput. The MTP catalog
validates the published MTP profiles.

### Next-token quality

Active MTP **does not change** the one-token, API-visible next-token
distribution. Mean BF16 KLD is **0.0731382442 nats** over the 32-prompt top-64
probe (P50 0.0492811974 / P95 0.1593743642 / Max 0.3348258511), top-1 29/32. The
active `draft-mtp` vs target-only graft next-token KLD is **0.0 (32/32 identical)**.

### Sequence-level behavior

A deterministic **8 prompt × 64 token** probe (`n_predict=64, temperature=-1, n_probs=64`) compares the same GGUF running active native `draft-mtp` against
target-only.

| Runtime | Client agg tok/s | Server decode tok/s | Draft acceptance | Exact 64-tok seqs | Token-position match |
| --- | --: | --- | --- | --: | --- |
| Target-only graft | 172.57 | ~210 (per-request timing) | n/a | 8/8 baseline | n/a |
| Fast active `draft-mtp`, `n_max=2` | 233.81 | 325.70 (/metrics) | 310/378 = 82.01%; per-position `(0.884, 0.747)` | 6/8 | 478/512 = 93.36% |
| `LLAMA_SPEC_VERIFY_SEQUENTIAL=1` | 164.34 | 204.88 (/metrics) | 313/375 = 83.47%; per-position `(0.884, 0.772)` | 5/8 | 455/512 = 88.87% |

[![MTP single-stream decode speedup on the deterministic 8x64 probe: target-only vs active MTP graft, client and server decode.](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/resolve/main/assets/06_mtp_tps.png)](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/assets/06_mtp_tps.png)

- Fast active MTP gives **~1.35× client throughput** (233.81 vs 172.57) and
server decode 325.70 vs ~210, with **82.01%** draft acceptance.
- It matches **6/8** sequences exactly and **478/512 = 93.36%** of token
positions; the two non-exact sequences first diverge at positions **60** and
**34**.
- The `LLAMA_SPEC_VERIFY_SEQUENTIAL=1` variant is both **slower** (164.34 /
204.88) **and** matches less (5/8, 88.87%; divergences at 25, 43, 60) — i.e. the
fast verifier is here both faster and more target-matching.

> **Note:** MTP self-speculation is **not bit-exact to target-only over long**
> **deterministic generations**. For workloads that require exact target
> reproduction, run target-only; for throughput on open-ended work, MTP wins.
> MTP is a single-user / low-concurrency win — it is not faster on saturated
> batches, which is why the adaptive throttle falls back to target-only.

### Sequence logprob comparability (API limitation)

llama.cpp omits candidate `top_logprobs` for most accepted speculative tokens, so
a naive "sequence KLD" looks **enormous** — that is an **API/logprob-availability**
**artifact, not model quality**. On the comparable subset where the emitted token
also matches, KLD is **≈ 0.001 nats** (near zero):

| Runtime | Paired positions | Comparable top-logprobs | Missing | Comparable same-token mean KLD (nats) | Max (nats) |
| --- | --: | --: | --: | --: | --: |
| Fast active `draft-mtp` vs target-only | 512 | 12 | 500 | 0.0011215739 | 0.0040239103 |
| Sequential verifier vs target-only | 512 | 10 | 502 | 0.0012293172 | 0.0040239103 |

Full sequence KLD cannot be computed without changing the runtime to return
target verifier logprobs for accepted speculative tokens.

### Draft-head distillation provenance

The meaningful KL-style metric for a standalone MTP draft head is **teacher KL**
**against cached target hidden states**, not standard base-model next-token KLD.
The draft head was trained and **measurably improved** across checkpoints —
teacher KL fell 1.92 → 1.50 → 0.76 and teacher argmax top-1 rose 0.54 → 0.69 →
0.89.

| Draft checkpoint | Tokens | Windows | Teacher KL | Teacher argmax top-1 | Teacher argmax top-5 | Gold top-1 |
| --- | --: | --: | --: | --: | --: | --: |
| `mtp-distill-kl-step500` | 22,637 | 128 | 1.9222025748 | 0.5360250917 | 0.7881786456 | 0.4425939833 |
| `snapshots-iq4-live-accepted-a/step-1000` | 58,754 | 413 | 1.5038724942 | 0.6876808388 | 0.8368451510 | 0.6764816013 |
| `snapshots-iq4-live-allrows-b/step-1000` | 32,264 | 2,048 | 0.7591610373 | 0.8908070915 | 0.9389722291 | 0.8076183982 |

[![MTP draft-head distillation progress: teacher KL falls (left) while top-1 agreement rises (right) across three checkpoints.](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/resolve/main/assets/05_mtp_draft_head_distill.png)](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/assets/05_mtp_draft_head_distill.png)

_Teacher KL is measured against cached target hidden states (not base next-token_
_KLD). These training artifacts live under `artifacts/mtp/` locally and are_
_referenced by name as provenance — they are not committed in this HF repo._

## Run with llama.cpp

Serve a body quant on GPU0 (`tp=1`):

```bash
QUANT=Q4_K_M PORT=8000 CTX_SIZE=8192 PARALLEL=16 REASONING=off \
  scripts/serve_llamacpp_gpu0.sh
```

Serve the integrated MTP graft for low-concurrency speculative decode (keep
adaptive throttling so saturated batches fall back to target-only):

```bash
QUANT=IQ4_XS-MTP-graft-headQ6 PORT=8000 CTX_SIZE=8192 PARALLEL=1 \
  CACHE_RAM=0 REASONING=off LLAMA_SPEC_MAX_DRAFTING_SLOTS=1 \
  scripts/serve_llamacpp_gpu0.sh
```

`REASONING=off` is the default because the model otherwise spends simple coding
prompts in `reasoning_content` before producing final `content`. See
[benchmarks/llamacpp-q4-reasoning-off-fix.md](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/benchmarks/llamacpp-q4-reasoning-off-fix.md).
Serving / quant / MTP profile catalogs:
[configs/serving\_profiles.yaml](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/configs/serving_profiles.yaml),
[configs/quant\_artifacts.yaml](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/configs/quant_artifacts.yaml),
[configs/mtp\_profiles.yaml](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/configs/mtp_profiles.yaml).

## Benchmark environment

**Hardware:** NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 97,887 MiB
VRAM, single GPU (`tp=1`). Driver 580.159.03, CUDA runtime reported by driver 13.0.

**Dedicated MTP KLD eval config (2026-06-28):** local llama.cpp CUDA server,
single **RTX PRO 6000 Blackwell 96GB**, `tp=1`, `CTX_SIZE=8192`, `PARALLEL=1`,
`CACHE_RAM=0`, `REASONING=off`.

- **llama.cpp build/commit:**`050ee92d04c2e1f639025786dea701c70e7d4204` (pinned for
the long-context TTFT run; earlier KLD/throughput sweeps were not separately pinned).
- **Reasoning mode:** all benchmarks in this card ran with `REASONING=off`.

## Provenance & reproducibility

- **Figures:** regenerate with [`scripts/make_charts.py`](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/scripts/make_charts.py)
→ `assets/02_fidelity_ladder.png`, `assets/03_throughput_tps.png`,
`assets/04_ttft_p95.png`, `assets/05_mtp_draft_head_distill.png`,
`assets/06_mtp_tps.png`.
- **Serving throughput / TTFT:** [benchmarks/llamacpp-quant-benchmarks.md](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/benchmarks/llamacpp-quant-benchmarks.md).
- **Body-quant fidelity:** [benchmarks/kld-quant-vs-bf16-top64.md](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/benchmarks/kld-quant-vs-bf16-top64.md).
- **Dedicated MTP KLD / sequence eval (2026-06-28):** [benchmarks/mtp-kld-eval-2026-06-28.md](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/benchmarks/mtp-kld-eval-2026-06-28.md)
and [benchmarks/mtp-dedicated-kld-evaluation.md](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/benchmarks/mtp-dedicated-kld-evaluation.md),
with raw evidence under [benchmarks/raw/](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/tree/main/benchmarks/raw/). Includes the
sequence-level probe and its logprob-availability limitation (full sequence KLD
is not computable from the current llama.cpp API response).
- **Adaptive MTP serving profile:** [benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md).
- **Long-context TTFT:** [benchmarks/llamacpp-longctx-ttft.md](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/blob/main/benchmarks/llamacpp-longctx-ttft.md).

Downloads last month14,974

GGUF

Model size

35B params

Architecture

qwen35moe

Chat template

Hardware compatibility

[Log In](https://huggingface.co/login?next=https%3A%2F%2Fhuggingface.co%2FLordNeel%2FOrnith-1.0-35B-GGUF-llamacpp-tp1) to add your hardware

3-bit

Q3\_K\_M

16.8 GB

4-bit

IQ4\_XS

19.6 GBMTPIQ4\_XS

18.9 GBQ4\_K\_M

21.2 GB

5-bit

Q5\_K\_M

24.7 GB

6-bit

Q6\_K

28.5 GB

8-bit

Q8\_0

36.9 GB

16-bit

MTPBF16

3.74 GB

Inference Providers [NEW](https://huggingface.co/docs/inference-providers)

[Text Generation](https://huggingface.co/tasks/text-generation "Learn more about text-generation")

This model isn't deployed by any Inference Provider. [🙋Ask for provider support](https://huggingface.co/spaces/huggingface/InferenceSupport/discussions/new?title=LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1&description=React%20to%20this%20comment%20with%20an%20emoji%20to%20vote%20for%20%5BLordNeel%2FOrnith-1.0-35B-GGUF-llamacpp-tp1%5D(%2FLordNeel%2FOrnith-1.0-35B-GGUF-llamacpp-tp1)%20to%20be%20supported%20by%20Inference%20Providers.%0A%0A(optional)%20Which%20providers%20are%20you%20interested%20in%3F%20(Novita%2C%20Hyperbolic%2C%20Together%E2%80%A6)%0A)

## Model tree for LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1

Base model

[deepreinforce-ai/Ornith-1.0-35B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B)

Quantized

( [109](https://huggingface.co/models?other=base_model:quantized:deepreinforce-ai/Ornith-1.0-35B))

this model

Includes a Multi-Token Prediction (MTP) head, which predicts several tokens at once to speed up inference.

Multi-Token Prediction (MTP) head: an auxiliary module that speeds up inference, not a standalone model.

StripeM-Inner

Inference providers allow you to run inference using different serverless providers.