[Skip to content](https://github.com/ggml-org/llama.cpp/issues/25096#start-of-content)

You signed in with another tab or window. [Reload](https://github.com/ggml-org/llama.cpp/issues/25096) to refresh your session.You signed out in another tab or window. [Reload](https://github.com/ggml-org/llama.cpp/issues/25096) to refresh your session.You switched accounts on another tab or window. [Reload](https://github.com/ggml-org/llama.cpp/issues/25096) to refresh your session.Dismiss alert

{{ message }}

### Uh oh!

There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/issues/25096).

[ggml-org](https://github.com/ggml-org)/ **[llama.cpp](https://github.com/ggml-org/llama.cpp)** Public

- [Notifications](https://github.com/login?return_to=%2Fggml-org%2Fllama.cpp) You must be signed in to change notification settings
- [Fork\\
20.2k](https://github.com/login?return_to=%2Fggml-org%2Fllama.cpp)
- [Star\\
119k](https://github.com/login?return_to=%2Fggml-org%2Fllama.cpp)


# Feature Request: DSpark confidence-scheduled verification & semi-autoregressive drafting\#25096

[New issue](https://github.com/login?return_to=https://github.com/ggml-org/llama.cpp/issues/25096)

Copy link

[New issue](https://github.com/login?return_to=https://github.com/ggml-org/llama.cpp/issues/25096)

Copy link

Open

Open

[Feature Request: DSpark confidence-scheduled verification & semi-autoregressive drafting](https://github.com/ggml-org/llama.cpp/issues/25096#top)#25096

Copy link

Labels

[enhancementNew feature or request](https://github.com/ggml-org/llama.cpp/issues?q=state%3Aopen%20label%3A%22enhancement%22) New feature or request

## Description

[![@gangula-karthik](https://avatars.githubusercontent.com/u/56480632?u=d4c9d0ae0b7812391b42ac8795337bf456cf4b8d&v=4&size=48)](https://github.com/gangula-karthik)

[gangula-karthik](https://github.com/gangula-karthik)

opened [1w agoon Jun 28, 2026](https://github.com/ggml-org/llama.cpp/issues/25096#issue-4761837238)

Last edited by gangula-karthik

Issue body actions

### Prerequisites

- I am running the latest code. Mention the version if possible as well.































I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md).































I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).































I reviewed the [Discussions](https://github.com/ggml-org/llama.cpp/discussions), and have a new and useful enhancement to share.




















To pick up a draggable item, press the space bar.
While dragging, use the arrow keys to move the item.
Press space again to drop the item in its new position, or press escape to cancel.



### Feature Description

This proposal adds two complementary enhancements to llama.cpp's speculative decoding pipeline, inspired by the DSpark paper ("DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation", DeepSeek-AI):

1. Confidence-Scheduled Verification. Today, the draft loop in `common/speculative.cpp` already performs per-token greedy early-stopping: it stops extending a draft as soon as a single token's top probability falls below `--spec-draft-p-min``(speculative.cpp:331)`. This proposal adds two things that don't currently exist:
   - Joint (cumulative) survival probability. Instead of comparing each token independently against a fixed threshold, accumulate a running product of the per-token draft probabilities and truncate the draft when the joint probability that the whole prefix survives falls below a threshold. For longer drafts this is stricter than the per-token check, since the product decays even when every individual token clears p\_min.
   - Load-aware verification scheduling. Tailor the verification length per request using the engine's current throughput/concurrency profile, not just draft-side confidence, so that under high concurrency, batch capacity isn't spent verifying low-survival tail tokens. llama.cpp currently has no load-aware component here.
2. Semi-Autoregressive (SAR) Draft Model Support (draft-sar). Add a new speculative type for SAR draft models, which pair a parallel backbone (full block proposal in one forward pass) with a lightweight sequential refinement module that introduces intra-block token dependencies.


### Motivation

Confidence-Scheduled Verification: The existing p\_min early-stop is a good per-token heuristic, but it makes each token's keep/drop decision in isolation. DSpark's insight is that what matters for wasted verification is the joint probability that an entire draft prefix survives, combined with how loaded the serving engine currently is. Under high concurrency, verifying tail tokens that are jointly unlikely to be accepted consumes target-model batch slots that could serve other requests. A cumulative-survival cutoff plus load-aware scheduling lets the system spend verification capacity where it pays off. DSpark reports this shifts the throughput-vs-latency Pareto frontier meaningfully in production serving. This change is inference-only and applies to all existing draft strategies (draft-simple, draft-eagle3, draft-mtp) with no new model weights.

SAR Draft Model Support: Parallel drafters (e.g. DFlash, MTP-style multi-token heads) lack inter-token dependencies, generate a whole block of draft tokens in one pass, treating each position independently. This causes suffix decay, acceptance rates fall sharply for later positions in the block because those tokens aren't conditioned on the earlier drafts in the same block. The SAR architecture adds a sequential refinement pass over the parallel backbone's output to model intra-block dependencies, directly attacking suffix decay. DSpark reports 60–85% per-user generation speedup over an MTP-1 baseline at matched throughput on live DeepSeek-V4 traffic. Supporting draft-sar gives llama.cpp users access to this where SAR draft heads are available or can be trained.

### Possible Implementation

Feature 1 - Confidence-Scheduled Verification:

- Add a parameter (e.g. --spec-draft-p-survival-min, float, default 0.0 = disabled) to common\_params\_speculative\_draft (common/common.h:323), alongside the existing p\_min/p\_split.
- In the draft loop in common/speculative.cpp (the section around speculative.cpp:304-351), maintain a running product of the accepted draft tokens' top probabilities (cur\_p->data\[0\].p is already available) and stop extending the draft when the product drops below the threshold — complementing, not replacing, the existing per-token p\_min check.
- For the load-aware component, the verification-length cap can be modulated by current batch occupancy / slot pressure in the server scheduler. This is the larger and more invasive part and could land as a follow-up after the joint-survival cutoff.
- This is inference-only and benefits all existing draft-model strategies.

Feature 2 - SAR Draft Model Support:

This follows the same pathway as EAGLE-3 (draft-eagle3):

- Add COMMON\_SPECULATIVE\_TYPE\_DRAFT\_SAR to the common\_speculative\_type enum.
- Implement a new speculative backend in common/speculative.cpp for the SAR two-pass drafting loop: (a) run parallel backbone in one forward pass to get a full draft block; (b) run the lightweight sequential refinement module over the block to refine token predictions with intra-block conditioning.
- Extend convert\_hf\_to\_gguf.py to support conversion of SAR draft head checkpoints from the DeepSpec format to GGUF.
- The SAR draft head architecture and open-source training code is available at: [https://github.com/deepseek-ai/DeepSpec](https://github.com/deepseek-ai/DeepSpec)

Reference: [DSpark paper](https://github.com/deepseek-ai/DeepSpec/blob/main/DSpark_paper.pdf)

Related llama.cpp work: EAGLE-3 ( [#18039](https://github.com/ggml-org/llama.cpp/pull/18039)), MTP heads.

👍React with 👍20Josephur, helloworld1983, 0xkeenz, ggppdk, Ankk98 and 15 more🚀React with 🚀2jerrydong1988 and sato20011

## Activity

[![](https://avatars.githubusercontent.com/u/56480632?s=64&u=d4c9d0ae0b7812391b42ac8795337bf456cf4b8d&v=4)gangula-karthik](https://github.com/gangula-karthik)

added

[enhancementNew feature or request](https://github.com/ggml-org/llama.cpp/issues?q=state%3Aopen%20label%3A%22enhancement%22) New feature or request

[1w agoon Jun 28, 2026](https://github.com/ggml-org/llama.cpp/issues/25096#event-27296830106)

### Ankk98 commented 5 days agoon Jun 28, 2026

[![@Ankk98](https://avatars.githubusercontent.com/u/24476799?u=b549739b02a749ef61fde958e36dd6411f9979ac&v=4&size=48)](https://github.com/Ankk98)

[Ankk98](https://github.com/Ankk98)

[5d agoon Jun 28, 2026](https://github.com/ggml-org/llama.cpp/issues/25096#issuecomment-4826856276)

More actions

In case this is useful, deekseek has released some weights for dspakr based models.

[https://huggingface.co/deepseek-ai/dspark\_gemma4\_12b\_block7](https://huggingface.co/deepseek-ai/dspark_gemma4_12b_block7)

🚀React with 🚀1jerrydong1988

### gangula-karthik commented 5 days agoon Jun 29, 2026

[![@gangula-karthik](https://avatars.githubusercontent.com/u/56480632?u=d4c9d0ae0b7812391b42ac8795337bf456cf4b8d&v=4&size=48)](https://github.com/gangula-karthik)

[gangula-karthik](https://github.com/gangula-karthik)

[5d agoon Jun 29, 2026](https://github.com/ggml-org/llama.cpp/issues/25096#issuecomment-4829301787)

Author

More actions

I'm working on my own fork for this currently, will link here as soon as I'm done

👍React with 👍1RyensX

### Ankk98 commented 5 days agoon Jun 29, 2026

[![@Ankk98](https://avatars.githubusercontent.com/u/24476799?u=b549739b02a749ef61fde958e36dd6411f9979ac&v=4&size=48)](https://github.com/Ankk98)

[Ankk98](https://github.com/Ankk98)

[5d agoon Jun 29, 2026](https://github.com/ggml-org/llama.cpp/issues/25096#issuecomment-4830535100)

More actions

I am trying to do a POC from last night but the results are not yet encouraging. Probably there are some issues. Working on a dspark speculative decoder implementation to make it work with gemma4 12b.

[https://github.com/Ankk98/llama.cpp/tree/ft-dspark](https://github.com/Ankk98/llama.cpp/tree/ft-dspark)

I will keep posting here for updates and results.

### Ankk98 commented 4 days agoon Jun 30, 2026

[![@Ankk98](https://avatars.githubusercontent.com/u/24476799?u=b549739b02a749ef61fde958e36dd6411f9979ac&v=4&size=48)](https://github.com/Ankk98)

[Ankk98](https://github.com/Ankk98)

[4d agoon Jun 30, 2026](https://github.com/ggml-org/llama.cpp/issues/25096#issuecomment-4840593741)

More actions

I am seeing 1.5X speedup over baseline model using dspark spec draft model for coding use.

- Baseline: Gemma 4 12B QAT Q4\_0 ~ 20-25 TPS TGP
- Gemma 4 12B QAT Q4\_0 + Dspark draft model Q4\_0 ~ 35-40 TPS TGP

### wjinxu commented 3 days agoon Jun 30, 2026

[![@wjinxu](https://avatars.githubusercontent.com/u/105263726?v=4&size=48)](https://github.com/wjinxu)

[wjinxu](https://github.com/wjinxu)

[3d agoon Jun 30, 2026](https://github.com/ggml-org/llama.cpp/issues/25096#issuecomment-4849421937)

More actions

Thanks for the detailed writeup. I've implemented the **semi-autoregressive drafting** half (Feature 2) in [#25173](https://github.com/ggml-org/llama.cpp/pull/25173).

In my tests it consistently beats DFlash across GSM8K/MATH500/HumanEval/MBPP/MT-Bench, across 4B/8B and bf16/Q8\_0/Q4\_K\_M targets — e.g. Qwen3-8B bf16, GSM8K: 79% vs 54% accept, ~3.8× vs ~2.8× over baseline.

For **confidence-scheduled verification** (Feature 1 — joint/cumulative survival + load-aware scheduling): the DSpark checkpoints already ship a trained confidence head, which [#25173](https://github.com/ggml-org/llama.cpp/pull/25173) loads but does not yet use. I'll follow up on wiring it once [#25173](https://github.com/ggml-org/llama.cpp/pull/25173) is merged.

### Ankk98 commented 2 days agoon Jul 2, 2026

[![@Ankk98](https://avatars.githubusercontent.com/u/24476799?u=b549739b02a749ef61fde958e36dd6411f9979ac&v=4&size=48)](https://github.com/Ankk98)

[Ankk98](https://github.com/Ankk98)

[2d agoon Jul 2, 2026](https://github.com/ggml-org/llama.cpp/issues/25096#issuecomment-4863173050)

More actions

I was trying to do a POC of dspark and interestingly parallel verification is leading to a divergence in output tokens as compared to vanilla model with same seed, zero temporature. Sequential verification worked fine but was slower. I will go deeper into this once I get some time.

[https://github.com/Ankk98/llama.cpp/tree/ft-dspark-qwen3](https://github.com/Ankk98/llama.cpp/tree/ft-dspark-qwen3)

[![](https://avatars.githubusercontent.com/u/14946854?s=64&u=2b1c450dd21c3bab62713eb7d6a10d02d79e8b27&v=4)rick-github](https://github.com/rick-github)

mentioned this [1d agoon Jul 3, 2026](https://github.com/ggml-org/llama.cpp/issues/25096#event-7962447122)

- [dspark option ollama/ollama#17016](https://github.com/ollama/ollama/issues/17016)


### kashif commented 7 hours agoon Jul 3, 2026

[![@kashif](https://avatars.githubusercontent.com/u/8100?v=4&size=48)](https://github.com/kashif)

[kashif](https://github.com/kashif)

[7h agoon Jul 3, 2026](https://github.com/ggml-org/llama.cpp/issues/25096#issuecomment-4879564687)

Contributor

More actions

see [#25173](https://github.com/ggml-org/llama.cpp/pull/25173)

[Sign up for free](https://github.com/signup?return_to=https://github.com/ggml-org/llama.cpp/issues/25096)**to join this conversation on GitHub.** Already have an account? [Sign in to comment](https://github.com/login?return_to=https://github.com/ggml-org/llama.cpp/issues/25096)

## Metadata

## Metadata

### Assignees

No one assigned

### Labels

[enhancementNew feature or request](https://github.com/ggml-org/llama.cpp/issues?q=state%3Aopen%20label%3A%22enhancement%22) New feature or request

### Type

No type

### Fields

No fields configured for issues without a type.

### Projects

No projects

### Milestone

No milestone

### Relationships

None yet

### Development

No branches or pull requests

### Participants

[![@kashif](https://avatars.githubusercontent.com/u/8100?s=64&v=4)](https://github.com/kashif)[![@Ankk98](https://avatars.githubusercontent.com/u/24476799?s=64&u=b549739b02a749ef61fde958e36dd6411f9979ac&v=4)](https://github.com/Ankk98)[![@gangula-karthik](https://avatars.githubusercontent.com/u/56480632?s=64&u=d4c9d0ae0b7812391b42ac8795337bf456cf4b8d&v=4)](https://github.com/gangula-karthik)[![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=64&v=4)](https://github.com/wjinxu)

## Issue actions

- ![](https://github.githubassets.com/assets/github-copilot-app-light-7138e992c731a2bb.png)Open in GitHub Copilot app

You can’t perform that action at this time.