[Skip to content](https://github.com/ggml-org/llama.cpp/pull/25173#start-of-content) You signed in with another tab or window. [Reload](https://github.com/ggml-org/llama.cpp/pull/25173) to refresh your session.You signed out in another tab or window. [Reload](https://github.com/ggml-org/llama.cpp/pull/25173) to refresh your session.You switched accounts on another tab or window. [Reload](https://github.com/ggml-org/llama.cpp/pull/25173) to refresh your session.Dismiss alert {{ message }} ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [ggml-org](https://github.com/ggml-org)/ **[llama.cpp](https://github.com/ggml-org/llama.cpp)** Public - [Notifications](https://github.com/login?return_to=%2Fggml-org%2Fllama.cpp) You must be signed in to change notification settings - [Fork\\ 20.2k](https://github.com/login?return_to=%2Fggml-org%2Fllama.cpp) - [Star\\ 119k](https://github.com/login?return_to=%2Fggml-org%2Fllama.cpp) ## Conversation [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=80&v=4)](https://github.com/wjinxu) ### ![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=48&v=4)**[wjinxu](https://github.com/wjinxu)** commented [4 days agoJun 30, 2026](https://github.com/ggml-org/llama.cpp/pull/25173\#issue-4777408490)• edited Loading \#\#\# Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). Copy link Copy Markdown This PR adds **DSpark** speculative decoding, layered on the merged **DFlash** drafter. DSpark (DeepSeek + PKU, 2026 — _"Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation"_, the DeepSpec repo) is DFlash plus a small **semi-autoregressive Markov head**: where DFlash takes an independent argmax at each block position (every position marginalizes over all possible predecessors, so acceptance decays along the block), DSpark adds a low-rank, previous-token-conditioned logit bias and samples the block left-to-right, so each draft conditions on the one actually sampled before it. This lifts accepted length at near-zero extra draft cost. DSpark **reuses the entire DFlash machinery unchanged** — the encoder/decoder graph, target-layer feature extraction (`llama_set_embeddings_layer_inp` / `_nextn`), KV-cache injection, and the verify/accept path. The only additions are: - a new draft architecture **`dspark`** (`llama_model_dspark : llama_model_dflash`) that reuses the DFlash graph and additionally loads the Markov head (`markov_w1`, `markov_w2`) and an optional confidence head; it shares the target's token-embeddings / lm\_head (same as DFlash); - a new speculative type **`draft-dspark`** (`common_speculative_impl_draft_dspark : common_speculative_impl_draft_dflash`) that reuses `process()` (extraction + injection) and overrides only `draft()`: the block is anchor-first (position 0 already predicts the first draft token) and sampled with the Markov bias `bias(prev) = markov_w2 · markov_w1[prev]`, computed on-device (`llama_dspark_markov_bias`); - a `Qwen3DSparkModel` converter. **Greedy decoding is lossless**: the Markov bias only changes _which_ tokens are proposed; every draft is still verified against the target, so the output is identical to non-speculative greedy. The confidence head is converted/loaded but **not used at inference** in this PR (phase 1); the draft-quality win from the Markov head is self-contained and is what the numbers below measure. ## How to run Complete example from scratch (Qwen3-8B). Drafts for other sizes are on the same org: `deepseek-ai/dspark_qwen3_{4b,8b,14b}_block7`. **1\. Get the models** — target + its DSpark draft: ``` huggingface-cli download Qwen/Qwen3-8B --local-dir Qwen3-8B huggingface-cli download deepseek-ai/dspark_qwen3_8b_block7 --local-dir dspark_qwen3_8b ``` **2\. Convert to GGUF** — the draft ships no tokenizer and reuses the target's, so pass `--target-model-dir`: ``` python convert_hf_to_gguf.py Qwen3-8B --outtype bf16 --outfile Qwen3-8B.gguf python convert_hf_to_gguf.py dspark_qwen3_8b --outtype bf16 \ --target-model-dir Qwen3-8B --outfile Qwen3-8B-DSpark.gguf ``` You may quantize the _target_ (e.g. `llama-quantize Qwen3-8B.gguf Qwen3-8B-Q4_K_M.gguf Q4_K_M`); keep the draft bf16 — it's tiny, and acceptance is unaffected by target quant. **3\. Build** with CUDA: ``` cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j ``` **4\. Run** — the only DSpark-specific flags are `-md ` and `--spec-type draft-dspark` (`--spec-draft-n-max` = draft tokens per step; the released checkpoints use block size 7): ``` ./build/bin/llama-server -m Qwen3-8B.gguf -md Qwen3-8B-DSpark.gguf \ --spec-type draft-dspark --spec-draft-n-max 7 \ --temp 0 --top-k 1 -np 1 -c 4096 -ngl 99 -fa on --jinja ``` **5\. Send a request** (the server logs `draft acceptance = ...` per request): ``` curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ "messages": [{"role": "user", "content": "Explain the Pythagorean theorem."}], "temperature": 0, "max_tokens": 256 }' ``` `llama-cli` works the same way (`-m ... -md ... --spec-type draft-dspark`). Note: `draft-dspark` needs the target's hidden states (KV-cache injection), so use `llama-server` / `llama-cli` — the `speculative-simple` example does not drive that path. ## Performance ### SpeedBench (llama.cpp's own `tools/server/bench/speed-bench`) Qwen3-8B (bf16), matched `--spec-draft-n-max 7`, `qualitative` split (11 categories), greedy. Baseline is the same server with no draft model. DSpark reaches **1.88× overall decode speedup** vs baseline (DFlash is 1.55×), and beats the merged DFlash on **every one of the 11 categories** (overall **1.21×**). DSpark vs baseline: ``` category base_avg_pred_t/s spec_avg_pred_t/s decode_speedup base_avg_latency spec_avg_latency latency_speedup accept_rate ------------- ----------------- ----------------- -------------- ---------------- ---------------- --------------- ----------- coding 58.16 123.57 2.12x 11.172s 5.458s 2.05x 0.3219 humanities 58.22 99.06 1.70x 9.573s 5.646s 1.70x 0.2340 math 58.21 109.86 1.89x 10.313s 5.409s 1.91x 0.2840 qa 58.23 107.32 1.84x 8.313s 4.486s 1.85x 0.2659 rag 57.91 123.54 2.13x 9.521s 4.639s 2.05x 0.3264 reasoning 58.21 99.29 1.71x 9.570s 5.622s 1.70x 0.2347 stem 58.19 98.92 1.70x 8.827s 5.205s 1.70x 0.2332 writing 57.82 111.32 1.93x 9.765s 5.282s 1.85x 0.2807 multilingual 58.18 121.96 2.10x 8.691s 4.250s 2.05x 0.3187 summarization 58.36 102.74 1.76x 5.309s 3.001s 1.77x 0.2530 roleplay 58.20 102.56 1.76x 14.139s 8.274s 1.71x 0.2454 overall 58.15 109.10 1.88x 9.563s 5.207s 1.84x 0.2698 ``` DSpark vs the merged DFlash (same `--spec-draft-n-max 7`): ``` category dflash_avg_pred_t/s dspark_avg_pred_t/s decode_speedup dflash_avg_latency dspark_avg_latency latency_speedup accept_rate ------------- ------------------- ------------------- -------------- ------------------ ------------------ --------------- ----------- coding 106.00 123.57 1.17x 6.343s 5.458s 1.16x 0.3219 humanities 83.61 99.06 1.18x 6.674s 5.646s 1.18x 0.2340 math 90.48 109.86 1.21x 6.529s 5.409s 1.21x 0.2840 qa 85.20 107.32 1.26x 5.650s 4.486s 1.26x 0.2659 rag 98.61 123.54 1.25x 5.733s 4.639s 1.24x 0.3264 reasoning 83.51 99.29 1.19x 6.681s 5.622s 1.19x 0.2347 stem 83.60 98.92 1.18x 6.154s 5.205s 1.18x 0.2332 writing 90.28 111.32 1.23x 6.443s 5.282s 1.22x 0.2807 multilingual 102.94 121.96 1.18x 5.016s 4.250s 1.18x 0.3187 summarization 85.85 102.74 1.20x 3.606s 3.001s 1.20x 0.2530 roleplay 79.96 102.56 1.28x 10.451s 8.274s 1.26x 0.2454 overall 90.00 109.10 1.21x 6.298s 5.207s 1.21x 0.2698 ``` Hardware: RTX 4090. Target Qwen/Qwen3-8B (bf16), draft deepseek-ai/dspark\_qwen3\_8b\_block7 (bf16). Greedy (`--temp 0 --top-k 1`), no-thinking, `--spec-draft-n-max 7`. Baseline = same llama-server with no draft model. DFlash is the merged drafter (`z-lab/Qwen3-8B-DFlash`, b16), run at the same matched draft size for an apples-to-apples comparison. Per-domain aggregate over the listed prompt counts. ### Losslessness Greedy decoding is lossless by construction (the draft is verified against the target). Output is coherent and matches non-speculative greedy. ### Qwen3-4B, target bf16 DSpark vs baseline (DFlash was not benchmarked at 4B — no nested-schema 4B DFlash draft available): | Domain | Baseline t/s | DSpark t/s (accept) | **DSpark** | | --- | --- | --- | --- | | GSM8K (40) | 103.1 | 354.0 (75.3%) | **3.43×** | | MATH500 (30) | 102.9 | 341.3 (71.7%) | **3.32×** | | HumanEval (30) | 103.9 | 340.0 (72.9%) | **3.27×** | | MBPP (30) | 103.6 | 281.4 (57.2%) | **2.72×** | | MT-Bench (30) | 102.8 | 190.4 (31.7%) | **1.85×** | | **geomean** | | | **2.85×** | ### Qwen3-8B, target bf16 | Domain | Baseline t/s | DFlash t/s (accept) | DSpark t/s (accept) | DFlash | **DSpark** | | --- | --- | --- | --- | --- | --- | | GSM8K (40) | 58.5 | 182.4 (53.7%) | 237.3 (78.9%) | 3.12× | **4.06×** | | MATH500 (30) | 58.5 | 195.7 (59.2%) | 223.2 (72.8%) | 3.35× | **3.82×** | | HumanEval (30) | 59.1 | 238.8 (77.2%) | 241.4 (81.7%) | 4.04× | **4.08×** | | MBPP (30) | 59.6 | 177.3 (53.3%) | 193.1 (63.7%) | 2.98× | **3.24×** | | MT-Bench (30) | 58.6 | 93.5 (19.7%) | 120.4 (31.3%) | 1.60× | **2.05×** | | **geomean** | | | | **2.89×** | **3.35×** | ### Qwen3-8B, target Q8\_0 | Domain | Baseline t/s | DFlash t/s (accept) | DSpark t/s (accept) | DFlash | **DSpark** | | --- | --- | --- | --- | --- | --- | | GSM8K (40) | 100.6 | 246.4 (53.2%) | 322.9 (77.8%) | 2.45× | **3.21×** | | MATH500 (30) | 100.5 | 266.2 (59.2%) | 305.7 (72.2%) | 2.65× | **3.04×** | | HumanEval (30) | 101.3 | 319.5 (76.5%) | 327.9 (81.4%) | 3.15× | **3.24×** | | MBPP (30) | 102.2 | 242.4 (54.3%) | 268.0 (64.3%) | 2.37× | **2.62×** | | MT-Bench (30) | 100.7 | 126.7 (19.3%) | 167.8 (31.4%) | 1.26× | **1.67×** | | **geomean** | | | | **2.28×** | **2.68×** | ### Qwen3-8B, target Q4\_K\_M | Domain | Baseline t/s | DFlash t/s (accept) | DSpark t/s (accept) | DFlash | **DSpark** | | --- | --- | --- | --- | --- | --- | | GSM8K (40) | 155.4 | 259.0 (52.9%) | 340.7 (77.4%) | 1.67× | **2.19×** | | MATH500 (30) | 155.2 | 284.9 (60.4%) | 326.1 (73.9%) | 1.84× | **2.10×** | | HumanEval (30) | 156.5 | 314.3 (71.1%) | 332.0 (78.4%) | 2.01× | **2.12×** | | MBPP (30) | 157.5 | 257.5 (55.5%) | 281.3 (66.0%) | 1.63× | **1.79×** | | MT-Bench (30) | 155.5 | 135.2 (19.7%) | 174.4 (30.6%) | 0.87× | **1.12×** | | **geomean** | | | | **1.54×** | **1.81×** | DSpark beats the merged DFlash on every domain (higher accept rate and higher throughput), for a **~1.16× geomean** speedup over DFlash. The gains are largest on reasoning (GSM8K +25pp accept, 1.30× over DFlash) and open chat (MT-Bench, 1.29×); on code (HumanEval) the two are close as both already accept ~80%. ## Future work - **Confidence head (phase 2):** wire the confidence-scheduled prefix pruning, with the paper's Sequential Temperature Scaling calibration. The big serving win in the paper comes from the batched scheduler, which is a separate, larger change. - **Markov-bias graph reuse:** the bias is computed as a tiny per-step ggml graph on the draft context's scheduler; building it once per block and re-running with new inputs would cut overhead. A fused bias+argmax kernel is a further option but would add a backend-specific op (the current path is pure ggml, no new operator). ## Requirements - I have read and agree with the [contributing guidelines](https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md) - AI usage disclosure: Yes, use Claude to help discuss and design the DSpark architecture, ask clarifying questions, and assist with writing tests. Everything remains under my control, and I reviewed every single line of AI-generated code. Sorry, something went wrong. ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). 👍5werewolfvit, egidijus, RytisLT, arsaboo, and cmp-nct reacted with thumbs up emoji😄1JamePeng reacted with laugh emoji❤️1kripper reacted with heart emoji🚀7tugucuk31, tsterbak, konstantinoskalyfommatos, stephanj, ALERTua, andrey-podko, and diegosouza reacted with rocket emoji👀4En3Tho, Ankk98, arsaboo, and mohammedalsayegh reacted with eyes emoji All reactions - 👍5 reactions - 😄1 reaction - ❤️1 reaction - 🚀7 reactions - 👀4 reactions [![@github-actions](https://avatars.githubusercontent.com/in/15368?s=40&v=4)](https://github.com/apps/github-actions)[github-actions](https://github.com/apps/github-actions) Bot added [model](https://github.com/ggml-org/llama.cpp/issues?q=state%3Aopen%20label%3Amodel) Model specific [conversion](https://github.com/ggml-org/llama.cpp/issues?q=state%3Aopen%20label%3Aconversion) labels [4 days agoJun 30, 2026](https://github.com/ggml-org/llama.cpp/pull/25173#event-27381581059) [![@ggml-gh-bot](https://avatars.githubusercontent.com/in/3131529?s=80&v=4)](https://github.com/apps/ggml-gh-bot) ### This comment was marked as resolved. [Sign in to view](https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fggml-org%2Fllama.cpp%2Fpull%2F25173) [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=40&v=4)](https://github.com/wjinxu)[wjinxu](https://github.com/wjinxu) [force-pushed](https://github.com/ggml-org/llama.cpp/compare/f3b83cdf22287a61a972c5ada9379b17d6d344b7..d74ff774222a2169a4b9897317e39198ef4765c3) the dspark-upstream branch from [`f3b83cd`](https://github.com/ggml-org/llama.cpp/commit/f3b83cdf22287a61a972c5ada9379b17d6d344b7) to [`d74ff77`](https://github.com/ggml-org/llama.cpp/commit/d74ff774222a2169a4b9897317e39198ef4765c3) [Compare](https://github.com/ggml-org/llama.cpp/compare/f3b83cdf22287a61a972c5ada9379b17d6d344b7..d74ff774222a2169a4b9897317e39198ef4765c3) [4 days agoJune 30, 2026 14:16](https://github.com/ggml-org/llama.cpp/pull/25173#event-27382550067) [![@github-actions](https://avatars.githubusercontent.com/in/15368?s=40&v=4)](https://github.com/apps/github-actions)[github-actions](https://github.com/apps/github-actions) Bot added the [testing](https://github.com/ggml-org/llama.cpp/issues?q=state%3Aopen%20label%3Atesting) Everything test related label [4 days agoJun 30, 2026](https://github.com/ggml-org/llama.cpp/pull/25173#event-27382572503) [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=40&v=4)](https://github.com/wjinxu)[wjinxu](https://github.com/wjinxu) [force-pushed](https://github.com/ggml-org/llama.cpp/compare/d74ff774222a2169a4b9897317e39198ef4765c3..37f2513db37ee18411a34c7a567f84c556e6c452) the dspark-upstream branch from [`d74ff77`](https://github.com/ggml-org/llama.cpp/commit/d74ff774222a2169a4b9897317e39198ef4765c3) to [`37f2513`](https://github.com/ggml-org/llama.cpp/commit/37f2513db37ee18411a34c7a567f84c556e6c452) [Compare](https://github.com/ggml-org/llama.cpp/compare/d74ff774222a2169a4b9897317e39198ef4765c3..37f2513db37ee18411a34c7a567f84c556e6c452) [4 days agoJune 30, 2026 14:39](https://github.com/ggml-org/llama.cpp/pull/25173#event-27383626631) [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=40&v=4)](https://github.com/wjinxu)[wjinxu](https://github.com/wjinxu) marked this pull request as ready for review [4 days agoJune 30, 2026 17:00](https://github.com/ggml-org/llama.cpp/pull/25173#event-27389926653) [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=40&v=4)](https://github.com/wjinxu)[wjinxu](https://github.com/wjinxu) requested review from a team, [CISC](https://github.com/CISC), [JohannesGaessler](https://github.com/JohannesGaessler) and [ggerganov](https://github.com/ggerganov) as [code owners](https://github.com/ggml-org/llama.cpp/blob/4f31eedb0ccf546b7e8d6bb243b170f12522f54d/CODEOWNERS#L25) [4 days agoJune 30, 2026 17:00](https://github.com/ggml-org/llama.cpp/pull/25173#event-27389927081) [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=80&v=4)](https://github.com/wjinxu) ### **[wjinxu](https://github.com/wjinxu)** commented [4 days agoJun 30, 2026](https://github.com/ggml-org/llama.cpp/pull/25173\#issuecomment-4846009082) Copy link Copy Markdown Author | | | --- | | Hi [@CISC](https://github.com/CISC) [@ggerganov](https://github.com/ggerganov) , this adds DSpark speculative decoding on top of the merged DFlash drafter ( [#22105](https://github.com/ggml-org/llama.cpp/pull/22105)). It's a small change — a new dspark draft arch and draft-dspark spec type that reuse DFlash's graph, feature extraction, KV-cache injection and verify path unchanged; the only new logic is the semi-autoregressive Markov head in draft(). Greedy stays lossless.
I benchmarked it against the merged DFlash using DeepSeek's released Qwen3 DSpark drafts. On Qwen3-8B at bf16 / Q8\_0 / Q4\_K\_M, DSpark beats DFlash on every domain (e.g. GSM8K bf16 4.06× vs 3.12×; full per-domain tables in the PR description).
I believe it's ready for review and I'm happy to walk through any part of it. | All reactions Sorry, something went wrong. ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [![@ruixiang63](https://avatars.githubusercontent.com/u/28504416?s=80&u=95e903edeba78031c75f993fdcd659c3d6aad3aa&v=4)](https://github.com/ruixiang63) ### **[ruixiang63](https://github.com/ruixiang63)** commented [4 days agoJun 30, 2026](https://github.com/ggml-org/llama.cpp/pull/25173\#issuecomment-4846049294)• edited Loading \#\#\# Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). Copy link Copy Markdown Member | | | --- | | Can you run SpeedBench to do the full comparison between DFlash and DSpark with the same `--spec-draft-n-max`? [https://github.com/ggml-org/llama.cpp/tree/master/tools/server/bench/speed-bench](https://github.com/ggml-org/llama.cpp/tree/master/tools/server/bench/speed-bench) | 👍1wjinxu reacted with thumbs up emoji All reactions - 👍1 reaction Sorry, something went wrong. ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [![@CISC](https://avatars.githubusercontent.com/u/1629204?s=40&v=4)](https://github.com/CISC)[CISC](https://github.com/CISC) requested a review from [ruixiang63](https://github.com/ruixiang63) [3 days agoJune 30, 2026 17:47](https://github.com/ggml-org/llama.cpp/pull/25173#event-27391819404) [![CISC](https://avatars.githubusercontent.com/u/1629204?s=60&v=4)](https://github.com/CISC) **[CISC](https://github.com/CISC)** reviewed [3 days agoJun 30, 2026](https://github.com/ggml-org/llama.cpp/pull/25173#pullrequestreview-4602810742) [View reviewed changes](https://github.com/ggml-org/llama.cpp/pull/25173/files) Comment thread[conversion/qwen.py](https://github.com/ggml-org/llama.cpp/pull/25173/files#diff-30f192da0dc9396d3b9e94af4540a43f6cd3d992eff36b3cce53a8359c2cf56f) Outdated Show resolvedHide resolved ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). Comment thread[conversion/qwen.py](https://github.com/ggml-org/llama.cpp/pull/25173/files#diff-30f192da0dc9396d3b9e94af4540a43f6cd3d992eff36b3cce53a8359c2cf56f) Outdated Show resolvedHide resolved ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [![CISC](https://avatars.githubusercontent.com/u/1629204?s=60&v=4)](https://github.com/CISC) **[CISC](https://github.com/CISC)** reviewed [3 days agoJun 30, 2026](https://github.com/ggml-org/llama.cpp/pull/25173#pullrequestreview-4602840717) [View reviewed changes](https://github.com/ggml-org/llama.cpp/pull/25173/files) Comment thread[conversion/qwen.py](https://github.com/ggml-org/llama.cpp/pull/25173/files#diff-30f192da0dc9396d3b9e94af4540a43f6cd3d992eff36b3cce53a8359c2cf56f) Outdated Show resolvedHide resolved ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=80&v=4)](https://github.com/wjinxu) ### **[wjinxu](https://github.com/wjinxu)** commented [3 days agoJun 30, 2026](https://github.com/ggml-org/llama.cpp/pull/25173\#issuecomment-4846549369) Copy link Copy Markdown Author | | | --- | | [@ruixiang63](https://github.com/ruixiang63) I've run the SpeedBench test set as you suggested, and updated the results in the PR description. DSpark does outperform DFlash across the board. | 👍2ruixiang63 and Ankk98 reacted with thumbs up emoji All reactions - 👍2 reactions Sorry, something went wrong. ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=40&v=4)](https://github.com/wjinxu)[wjinxu](https://github.com/wjinxu) [force-pushed](https://github.com/ggml-org/llama.cpp/compare/aae2941a6962548475b95437912a6a799e8f4087..d8b38f23ec3fd6175a95acbf7a2f17c1f7a1eb83) the dspark-upstream branch 2 times, most recently from [`aae2941`](https://github.com/ggml-org/llama.cpp/commit/aae2941a6962548475b95437912a6a799e8f4087) to [`d8b38f2`](https://github.com/ggml-org/llama.cpp/commit/d8b38f23ec3fd6175a95acbf7a2f17c1f7a1eb83) [Compare](https://github.com/ggml-org/llama.cpp/compare/aae2941a6962548475b95437912a6a799e8f4087..d8b38f23ec3fd6175a95acbf7a2f17c1f7a1eb83) [3 days agoJune 30, 2026 18:36](https://github.com/ggml-org/llama.cpp/pull/25173#event-27393714944) [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=40&v=4)](https://github.com/wjinxu)[wjinxu](https://github.com/wjinxu) mentioned this pull request [3 days agoJul 1, 2026](https://github.com/ggml-org/llama.cpp/pull/25173#ref-issue-4761837238) [Feature Request: DSpark confidence-scheduled verification & semi-autoregressive drafting\\ #25096](https://github.com/ggml-org/llama.cpp/issues/25096) Open 4 tasks [![@nipeone](https://avatars.githubusercontent.com/u/4449628?s=80&u=343535dc008e99c4ebe25adbce50e66a37ba6d30&v=4)](https://github.com/nipeone) ### **[nipeone](https://github.com/nipeone)** commented [3 days agoJul 1, 2026](https://github.com/ggml-org/llama.cpp/pull/25173\#issuecomment-4850307620) Copy link Copy Markdown | | | --- | | could you give some examples how to use? | 👀1wjinxu reacted with eyes emoji All reactions - 👀1 reaction Sorry, something went wrong. ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=80&v=4)](https://github.com/wjinxu) ### **[wjinxu](https://github.com/wjinxu)** commented [3 days agoJul 1, 2026](https://github.com/ggml-org/llama.cpp/pull/25173\#issuecomment-4850429456) Copy link Copy Markdown Author | | | --- | | > could you give some examples how to use?
Good point — I've updated the PR description with a more detailed, copy-pasteable end-to-end example (download → convert → build → run → curl). Let me know if anything's unclear. | All reactions Sorry, something went wrong. ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [![lym000000](https://avatars.githubusercontent.com/u/32905510?s=60&v=4)](https://github.com/lym000000) **[lym000000](https://github.com/lym000000)** reviewed [3 days agoJul 1, 2026](https://github.com/ggml-org/llama.cpp/pull/25173#pullrequestreview-4605922293) [View reviewed changes](https://github.com/ggml-org/llama.cpp/pull/25173/files) Comment thread[src/llama-model.cpp](https://github.com/ggml-org/llama.cpp/pull/25173/files#diff-36e262e316ec1404e29880eb8b8ce4660ac584f0d0434710efc48a66497bdb59) Outdated Show resolvedHide resolved ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=40&v=4)](https://github.com/wjinxu)[wjinxu](https://github.com/wjinxu) [force-pushed](https://github.com/ggml-org/llama.cpp/compare/d8b38f23ec3fd6175a95acbf7a2f17c1f7a1eb83..47f3442043f86c748e5774f9a899b137f990a761) the dspark-upstream branch from [`d8b38f2`](https://github.com/ggml-org/llama.cpp/commit/d8b38f23ec3fd6175a95acbf7a2f17c1f7a1eb83) to [`47f3442`](https://github.com/ggml-org/llama.cpp/commit/47f3442043f86c748e5774f9a899b137f990a761) [Compare](https://github.com/ggml-org/llama.cpp/compare/d8b38f23ec3fd6175a95acbf7a2f17c1f7a1eb83..47f3442043f86c748e5774f9a899b137f990a761) [3 days agoJuly 1, 2026 05:17](https://github.com/ggml-org/llama.cpp/pull/25173#event-27411196914) [![@am17an](https://avatars.githubusercontent.com/u/2929750?s=80&u=2858e37ff0a8e7d63ee1df78115001c3ec93ed43&v=4)](https://github.com/am17an) ### **[am17an](https://github.com/am17an)** commented [3 days agoJul 1, 2026](https://github.com/ggml-org/llama.cpp/pull/25173\#issuecomment-4850908620) Copy link Copy Markdown Contributor | | | --- | | DSV4 support was merged in [#24162](https://github.com/ggml-org/llama.cpp/pull/24162), ideally this PR should cover that model as well and try to replicate a similar speedup | All reactions Sorry, something went wrong. ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=80&v=4)](https://github.com/wjinxu) ### **[wjinxu](https://github.com/wjinxu)** commented [3 days agoJul 1, 2026](https://github.com/ggml-org/llama.cpp/pull/25173\#issuecomment-4851011331) Copy link Copy Markdown Author | | | --- | | > DSV4 support was merged in [#24162](https://github.com/ggml-org/llama.cpp/pull/24162), ideally this PR should cover that model as well and try to replicate a similar speedup
Thanks! DeepSeek hasn't open-sourced the DSpark weights for DeepSeek-V4 though — only the Qwen3 and Gemma4 drafts are released. So this PR covers Qwen3 for now, and I'll add Gemma4 as a small follow-up. | All reactions Sorry, something went wrong. ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [![@am17an](https://avatars.githubusercontent.com/u/2929750?s=80&u=2858e37ff0a8e7d63ee1df78115001c3ec93ed43&v=4)](https://github.com/am17an) ### **[am17an](https://github.com/am17an)** commented [3 days agoJul 1, 2026](https://github.com/ggml-org/llama.cpp/pull/25173\#issuecomment-4851215028) Copy link Copy Markdown Contributor | | | --- | | I think they're a part of the spec decoding module [https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark) i.e not distributed separately | 👍1wjinxu reacted with thumbs up emoji All reactions - 👍1 reaction Sorry, something went wrong. ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=80&v=4)](https://github.com/wjinxu) ### **[wjinxu](https://github.com/wjinxu)** commented [3 days agoJul 1, 2026](https://github.com/ggml-org/llama.cpp/pull/25173\#issuecomment-4851394608) Copy link Copy Markdown Author | | | --- | | > I think they're a part of the spec decoding module [https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark) i.e not distributed separately
Sorry, and thanks for the heads-up. For this PR I'd like to keep the scope a bit narrower for now - land the Qwen3 DSpark path first and get it solid, then add Gemma4 and DSV4 as follow-ups. Does that sound ok? | All reactions Sorry, something went wrong. ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [![@am17an](https://avatars.githubusercontent.com/u/2929750?s=80&u=2858e37ff0a8e7d63ee1df78115001c3ec93ed43&v=4)](https://github.com/am17an) ### **[am17an](https://github.com/am17an)** commented [3 days agoJul 1, 2026](https://github.com/ggml-org/llama.cpp/pull/25173\#issuecomment-4851480421) Copy link Copy Markdown Contributor | | | --- | | Okay, will try to review. From a cursory look it does not look like adding `llama_dspark*` is the right thing to do. Honestly don't have a good feeling about the PR, too much AI code. | 👍1wjinxu reacted with thumbs up emoji All reactions - 👍1 reaction Sorry, something went wrong. ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=80&v=4)](https://github.com/wjinxu) ### **[wjinxu](https://github.com/wjinxu)** commented [3 days agoJul 1, 2026](https://github.com/ggml-org/llama.cpp/pull/25173\#issuecomment-4852115845) Copy link Copy Markdown Author | | | --- | | > Okay, will try to review. From a cursory look it does not look like adding `llama_dspark*` is the right thing to do. Honestly don't have a good feeling about the PR, too much AI code.
I agree that `llama_dspark_*` shouldn't be part of the API. The issue is that the Markov bias computation `(W2 @ W1[prev])` needs the draft weights and has to run on the backend - I measured it, and doing it on the host is too slow. But the DSpark drafter lives in `common/`, which can't reach the draft weights, so once the API is removed there's no way to trigger that computation from `common/`. | All reactions Sorry, something went wrong. ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=40&v=4)](https://github.com/wjinxu)[wjinxu](https://github.com/wjinxu) marked this pull request as draft [2 days agoJuly 2, 2026 02:24](https://github.com/ggml-org/llama.cpp/pull/25173#event-27457788850) [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=40&v=4)](https://github.com/wjinxu) ` spec: add DSpark speculative decoding ` … Loading Loading status checks… ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). ` 96c5be9 ` ``` DSpark (DeepSpec, 2026) on top of the merged DFlash drafter. It reuses the DFlash encoder/decoder graph, target feature extraction and KV-cache injection, and the verify/accept path unchanged; the draft model is a new "dspark" arch adding a low-rank Markov head (markov_w1/w2) and an optional (unused here) confidence head. No new public APIs. The proposal is the only change: the block is anchor-first (position 0 already predicts the first draft) and the decoder graph applies a semi-autoregressive, previous-token conditioned logit bias in-graph, chained per block position: logits'(i) = logits(i) + markov_w2 . markov_w1[prev(i)] prev(0) = the block's anchor token, prev(i>0) = argmax(logits'(i-1)) vectorized across all blocks in the batch; the anchors are fed through a dedicated graph input (token 0 of every block). Greedy stays lossless (verify unchanged, same as DFlash). - new arch "dspark" (llama_model_dspark : llama_model_dflash, reuses the graph, loads the markov/confidence tensors; shares the target's embed/lm_head). - Qwen3DSparkModel converter. - new spec type "draft-dspark" (common_speculative_impl_draft_dspark : common_speculative_impl_draft_dflash, overrides draft() only: submits whole anchor-first blocks and greedily reads back the biased logits). ``` [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=40&v=4)](https://github.com/wjinxu)[wjinxu](https://github.com/wjinxu) [force-pushed](https://github.com/ggml-org/llama.cpp/compare/47f3442043f86c748e5774f9a899b137f990a761..96c5be9dd0fb9971b09df9e2b0647c749aa4fb0d) the dspark-upstream branch from [`47f3442`](https://github.com/ggml-org/llama.cpp/commit/47f3442043f86c748e5774f9a899b137f990a761) to [`96c5be9`](https://github.com/ggml-org/llama.cpp/commit/96c5be9dd0fb9971b09df9e2b0647c749aa4fb0d) [Compare](https://github.com/ggml-org/llama.cpp/compare/47f3442043f86c748e5774f9a899b137f990a761..96c5be9dd0fb9971b09df9e2b0647c749aa4fb0d) [2 days agoJuly 2, 2026 07:24](https://github.com/ggml-org/llama.cpp/pull/25173#event-27466060221) [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=80&v=4)](https://github.com/wjinxu) ### **[wjinxu](https://github.com/wjinxu)** commented [2 days agoJul 2, 2026](https://github.com/ggml-org/llama.cpp/pull/25173\#issuecomment-4863309632) Copy link Copy Markdown Author | | | --- | | [@am17an](https://github.com/am17an) Thanks for the feedback — you were right about the API. I've reworked the PR:
All llama\_dspark\_\* public APIs are gone. The Markov head is now applied inside the dspark decoder graph(chained per block position, vectorized across blocks), so llama.h / llama-ext.h are untouched and common/speculative.cpp only overrides draft() on top of the DFlash impl, same pattern as before.
I've reviewed and can explain every line of the code myself, and re-verified.
I'd really appreciate another look when you have time. | ❤️7stephanj, Farenheith, egidijus, andrey-podko, therealkenc, podium868909, and IIIIIllllIIIIIlllll reacted with heart emoji All reactions - ❤️7 reactions Sorry, something went wrong. ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=40&v=4)](https://github.com/wjinxu)[wjinxu](https://github.com/wjinxu) marked this pull request as ready for review [2 days agoJuly 2, 2026 07:34](https://github.com/ggml-org/llama.cpp/pull/25173#event-27466448317) [![am17an](https://avatars.githubusercontent.com/u/2929750?s=60&v=4)](https://github.com/am17an) **[am17an](https://github.com/am17an)** reviewed [19 minutes agoJul 4, 2026](https://github.com/ggml-org/llama.cpp/pull/25173#pullrequestreview-4628915880) [View reviewed changes](https://github.com/ggml-org/llama.cpp/pull/25173/files/96c5be9dd0fb9971b09df9e2b0647c749aa4fb0d) Comment thread[common/speculative.cpp](https://github.com/ggml-org/llama.cpp/pull/25173/files/96c5be9dd0fb9971b09df9e2b0647c749aa4fb0d#diff-2c8c834a4f5b2c394b1a26e7b6f66e87682f7f4bf348d4b12a83dc9a977be5b1) Comment on lines +1203 to +1207 | | | | | --- | --- | --- | | | | char buf\[32\] = {}; | | | | if (llama\_model\_meta\_val\_str(model\_dft, "dspark.block\_size", buf, sizeof(buf)) < 0) { | | | | GGML\_ABORT("DSpark: missing required metadata key 'dspark.block\_size'"); | | | | } | | | | block\_size = std::atoi(buf); | ### ![@am17an](https://avatars.githubusercontent.com/u/2929750?s=48&v=4)**[am17an](https://github.com/am17an)** [22 minutes agoJul 4, 2026](https://github.com/ggml-org/llama.cpp/pull/25173\#discussion_r3522599238) Copy link Copy Markdown Contributor There was a problem hiding this comment. ### Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. [Learn more](https://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment). Choose a reason SpamAbuseOff TopicOutdatedDuplicateResolvedLow QualityHide comment block size exists in `common_speculative_impl_draft_dflash`, add it there Sorry, something went wrong. ### Uh oh! There was an error while loading. [Please reload this page](https://github.com/ggml-org/llama.cpp/pull/25173). All reactions This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. [Learn more about bidirectional Unicode characters](https://github.co/hiddenchars) [Show hidden characters](https://github.com/ggml-org/llama.cpp/pull/25173) [Sign up for free](https://github.com/join?source=comment-repo) **to join this conversation on GitHub**. Already have an account? [Sign in to comment](https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fggml-org%2Fllama.cpp%2Fpull%2F25173) ### Reviewers [![@CISC](https://avatars.githubusercontent.com/u/1629204?s=40&v=4)](https://github.com/CISC)[CISC](https://github.com/CISC)CISC left review comments [![@am17an](https://avatars.githubusercontent.com/u/2929750?s=40&v=4)](https://github.com/am17an)[am17an](https://github.com/am17an)am17an left review comments [![@JohannesGaessler](https://avatars.githubusercontent.com/u/18492268?s=40&v=4)](https://github.com/JohannesGaessler)[JohannesGaessler](https://github.com/JohannesGaessler)Awaiting requested review from JohannesGaesslerJohannesGaessler is a code owner [![@ggerganov](https://avatars.githubusercontent.com/u/1991296?s=40&v=4)](https://github.com/ggerganov)[ggerganov](https://github.com/ggerganov)Awaiting requested review from ggerganovggerganov is a code owner [![@ruixiang63](https://avatars.githubusercontent.com/u/28504416?s=40&v=4)](https://github.com/ruixiang63)[ruixiang63](https://github.com/ruixiang63)Awaiting requested review from ruixiang63 +1 more reviewer [![@lym000000](https://avatars.githubusercontent.com/u/32905510?s=40&v=4)](https://github.com/lym000000)[lym000000](https://github.com/lym000000)lym000000 left review comments Reviewers whose approvals may not affect merge requirements At least 2 approving reviews are required to merge this pull request. ### Assignees No one assigned ### Labels [conversion](https://github.com/ggml-org/llama.cpp/issues?q=state%3Aopen%20label%3Aconversion) [model](https://github.com/ggml-org/llama.cpp/issues?q=state%3Aopen%20label%3Amodel) Model specific [testing](https://github.com/ggml-org/llama.cpp/issues?q=state%3Aopen%20label%3Atesting) Everything test related ### Projects None yet ### Milestone No milestone ### Development Successfully merging this pull request may close these issues. None yet ### 6 participants [![@wjinxu](https://avatars.githubusercontent.com/u/105263726?s=52&v=4)](https://github.com/wjinxu)[![@ruixiang63](https://avatars.githubusercontent.com/u/28504416?s=52&v=4)](https://github.com/ruixiang63)[![@nipeone](https://avatars.githubusercontent.com/u/4449628?s=52&v=4)](https://github.com/nipeone)[![@am17an](https://avatars.githubusercontent.com/u/2929750?s=52&v=4)](https://github.com/am17an)[![@CISC](https://avatars.githubusercontent.com/u/1629204?s=52&v=4)](https://github.com/CISC)[![@lym000000](https://avatars.githubusercontent.com/u/32905510?s=52&v=4)](https://github.com/lym000000) Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later. You can’t perform that action at this time.