![](https://huggingface.co/avatars/c21bc71296e7ad0b395ff24774be379f.svg)[LordNeel](https://huggingface.co/LordNeel) commited on 3 days ago

Commit

c920ea4

·

verified·

1
Parent(s): [532b5f8](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/commit/532b5f80c7b536ab2d4392323f50ca9be4b6cd59)

# Add SWE-bench quant and reasoning metrics

[Browse files](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/tree/c920ea497b468e9fb3b7e1e52931a7f31ad925bf)

Files changed (11)hideshow

01. [MODEL\_RELEASE\_REPORT.md](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/commit/c920ea497b468e9fb3b7e1e52931a7f31ad925bf#d2h-330243)+450-0
02. [README.md](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/commit/c920ea497b468e9fb3b7e1e52931a7f31ad925bf#d2h-611991)+407-240
03. [benchmarks/raw/swebench/ornith-q6-think-verified100-summary.json](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/commit/c920ea497b468e9fb3b7e1e52931a7f31ad925bf#d2h-835044)+67-0
04. [benchmarks/raw/swebench/ornith\_iq4\_xs\_mtp\_graft\_verified\_mini.json](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/commit/c920ea497b468e9fb3b7e1e52931a7f31ad925bf#d2h-594243)+170-0
05. [benchmarks/raw/swebench/ornith\_iq4\_xs\_verified\_mini.json](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/commit/c920ea497b468e9fb3b7e1e52931a7f31ad925bf#d2h-913044)+167-0
06. [benchmarks/raw/swebench/ornith\_q3\_k\_m\_verified\_mini.json](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/commit/c920ea497b468e9fb3b7e1e52931a7f31ad925bf#d2h-701250)+169-0
07. [benchmarks/raw/swebench/ornith\_q4\_k\_m\_verified\_mini.json](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/commit/c920ea497b468e9fb3b7e1e52931a7f31ad925bf#d2h-708003)+170-0
08. [benchmarks/raw/swebench/ornith\_q5\_k\_m\_verified\_mini.json](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/commit/c920ea497b468e9fb3b7e1e52931a7f31ad925bf#d2h-714756)+171-0
09. [benchmarks/raw/swebench/ornith\_q6\_k\_verified\_mini.json](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/commit/c920ea497b468e9fb3b7e1e52931a7f31ad925bf#d2h-411209)+170-0
10. [benchmarks/raw/swebench/ornith\_q8\_0\_verified\_mini.json](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/commit/c920ea497b468e9fb3b7e1e52931a7f31ad925bf#d2h-091426)+169-0
11. [benchmarks/swebench-agent-evals.md](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1/commit/c920ea497b468e9fb3b7e1e52931a7f31ad925bf#d2h-231781)+75-0

MODEL\_RELEASE\_REPORT.mdADDED
Viewed

|     |     |
| --- | --- |
|  | @@ -0,0 +1,450 @@ |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

|     |     |
| --- | --- |
|  |  |
| 1 | +\# Ornith-1.0-35B GGUF Release Report |
| 2 | + |
| 3 | +Date: 2026-06-28 |
| 4 | + |
| 5 | +Hardware for reported GPU numbers: one NVIDIA RTX PRO 6000 Blackwell Max-Q |
| 6 | +96GB, \`tp=1\`, llama.cpp CUDA server. The release policy is single-GPU serving |
| 7 | +only: one model copy per GPU, no tensor-parallel serving profile. |
| 8 | + |
| 9 | +\## Executive Summary |
| 10 | + |
| 11 | +This release turns Ornith-1.0-35B into a practical llama.cpp GGUF stack with |
| 12 | +validated serving profiles, locally produced smaller quants, quantitative KLD |
| 13 | +quality checks, and a working native MTP path. |
| 14 | + |
| 15 | +Recommended use: |
| 16 | + |
| 17 | +\| Use case \| Artifact \| Why \| |
| 18 | +\|---\|---\|---\| |
| 19 | +\| Default serving speed \| \`ornith-1.0-35b-Q4\_K\_M.gguf\` \| Fastest validated c16 profile, 655.6 tok/s, 21.31 GiB loaded VRAM \| |
| 20 | +\| Lowest memory \| \`ornith-1.0-35b-Q3\_K\_M.gguf\` \| 15.61 GiB on disk, 17.27 GiB loaded VRAM, behavior gate passed \| |
| 21 | +\| Middle footprint \| \`ornith-1.0-35b-IQ4\_XS.gguf\` \| 17.64 GiB on disk, lower KLD drift than Q3\_K\_M \| |
| 22 | +\| Quality/footprint \| \`ornith-1.0-35b-Q6\_K.gguf\` \| 0.0165 mean top-64 KLD nats and 32/32 top-1 match \| |
| 23 | +\| Native low-concurrency MTP \| \`ornith-1.0-35b-IQ4\_XS-MTP-graft-headQ6.gguf\` \| c1 MTP speedup with high acceptance; adaptive policy avoids saturated c16 loss \| |
| 24 | + |
| 25 | +Key result: Q4\_K\_M is the production default. The integrated IQ4\_XS-MTP graft is |
| 26 | +the MTP release candidate for low-concurrency/single-user experiments. Always-on |
| 27 | +MTP is not recommended for saturated c16 server traffic because target-only |
| 28 | +decoding already saturates the GPU more efficiently. |
| 29 | + |
| 30 | +\## Quantization Work |
| 31 | + |
| 32 | +\### What Was Built |
| 33 | + |
| 34 | +Two main-body GGUF quants were produced locally from the upstream BF16 GGUF: |
| 35 | + |
| 36 | +\| Artifact \| Quant \| Quantization detail \| |
| 37 | +\|---\|---\|---\| |
| 38 | +\| \`ornith-1.0-35b-Q3\_K\_M.gguf\` \| Q3\_K\_M \| \`llama-quantize ornith-1.0-35b-bf16.gguf ornith-1.0-35b-Q3\_K\_M.gguf Q3\_K\_M\`; source 66,152.24 MiB / 16.01 BPW, output 15,977.65 MiB / 3.87 BPW \| |
| 39 | +\| \`ornith-1.0-35b-IQ4\_XS.gguf\` \| IQ4\_XS \| \`llama-quantize ornith-1.0-35b-bf16.gguf ornith-1.0-35b-IQ4\_XS.gguf IQ4\_XS 32\`; no importance matrix; source 66,152.24 MiB / 16.01 BPW, output 18,051.46 MiB / 4.37 BPW \| |
| 40 | + |
| 41 | +The existing upstream GGUFs were mirrored and validated: |
| 42 | + |
| 43 | +\| Artifact \| Quant \| Release status \| |
| 44 | +\|---\|---\|---\| |
| 45 | +\| \`ornith-1.0-35b-Q4\_K\_M.gguf\` \| Q4\_K\_M \| validated, production default \| |
| 46 | +\| \`ornith-1.0-35b-Q5\_K\_M.gguf\` \| Q5\_K\_M \| validated, higher-quality mid option \| |
| 47 | +\| \`ornith-1.0-35b-Q6\_K.gguf\` \| Q6\_K \| validated, quality/footprint option \| |
| 48 | +\| \`ornith-1.0-35b-Q8\_0.gguf\` \| Q8\_0 \| validated, closest to BF16 by mean KLD \| |
| 49 | + |
| 50 | +MTP-specific artifacts were also produced or published: |
| 51 | + |
| 52 | +\| Artifact \| Type \| Purpose \| |
| 53 | +\|---\|---\|---\| |
| 54 | +\| \`ornith-1.0-35b-IQ4\_XS-MTP-graft-headQ6.gguf\` \| integrated target + MTP block \| native \`draft-mtp\` low-concurrency profile \| |
| 55 | +\| \`ornith-1.0-35b-mtp-bootstrap-layer39-IQ4\_XS.gguf\` \| bootstrap external draft \| proved MTP plumbing, not recommended \| |
| 56 | +\| \`ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q6\_K.gguf\` \| trained external draft \| MTP development and regression testing \| |
| 57 | +\| \`ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q5\_K\_M.gguf\` \| trained external draft \| smaller draft variant \| |
| 58 | +\| \`ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-IQ4\_XS.gguf\` \| trained external draft \| smallest trained draft, lower acceptance \| |
| 59 | + |
| 60 | +\### Footprint Chart |
| 61 | + |
| 62 | +Loaded VRAM is from the short-context throughput profile |
| 63 | +\`CTX\_SIZE=8192 PARALLEL=16\`. |
| 64 | + |
| 65 | +\| Quant \| GGUF GiB \| Loaded VRAM GiB \| Disk vs Q4\_K\_M \| VRAM vs Q4\_K\_M \| |
| 66 | +\|---\|---:\|---:\|---:\|---:\| |
| 67 | +\| Q3\_K\_M \| 15.61 \| 17.27 \| -20.8% \| -19.0% \| |
| 68 | +\| IQ4\_XS \| 17.64 \| 19.34 \| -10.5% \| -9.2% \| |
| 69 | +\| Q4\_K\_M \| 19.71 \| 21.31 \| baseline \| baseline \| |
| 70 | +\| Q5\_K\_M \| 23.03 \| 24.65 \| +16.8% \| +15.7% \| |
| 71 | +\| Q6\_K \| 26.56 \| 28.03 \| +34.8% \| +31.5% \| |
| 72 | +\| Q8\_0 \| 34.37 \| 35.72 \| +74.4% \| +67.6% \| |
| 73 | + |
| 74 | +Footprint bar, loaded VRAM: |
| 75 | + |
| 76 | +\| Quant \| Loaded VRAM chart \| |
| 77 | +\|---\|---\| |
| 78 | +\| Q3\_K\_M \| \`################# 17.27 GiB\` \| |
| 79 | +\| IQ4\_XS \| \`################### 19.34 GiB\` \| |
| 80 | +\| Q4\_K\_M \| \`##################### 21.31 GiB\` \| |
| 81 | +\| Q5\_K\_M \| \`######################### 24.65 GiB\` \| |
| 82 | +\| Q6\_K \| \`############################ 28.03 GiB\` \| |
| 83 | +\| Q8\_0 \| \`#################################### 35.72 GiB\` \| |
| 84 | + |
| 85 | +\### Throughput Chart |
| 86 | + |
| 87 | +Aggregate output tokens/second from \`scripts/bench\_openai.py --stream |
| 88 | +--max-tokens 256\`; all rows had 0 failed requests. |
| 89 | + |
| 90 | +\| Quant \| c1 tok/s \| c4 tok/s \| c8 tok/s \| c16 tok/s \| c16 p95 TTFT ms \| |
| 91 | +\|---\|---:\|---:\|---:\|---:\|---:\| |
| 92 | +\| Q3\_K\_M \| 240.5 \| 422.0 \| 464.4 \| 493.0 \| 493.7 \| |
| 93 | +\| IQ4\_XS \| 234.1 \| 297.7 \| 411.5 \| 476.0 \| 541.5 \| |
| 94 | +\| Q4\_K\_M \| 243.3 \| 458.3 \| 615.0 \| 655.6 \| 650.0 \| |
| 95 | +\| Q5\_K\_M \| 236.7 \| 311.0 \| 439.0 \| 638.6 \| 620.4 \| |
| 96 | +\| Q6\_K \| 225.9 \| 295.8 \| 409.6 \| 603.3 \| 657.4 \| |
| 97 | +\| Q8\_0 \| 208.5 \| 281.5 \| 405.8 \| 601.4 \| 725.8 \| |
| 98 | + |
| 99 | +c16 throughput bar: |
| 100 | + |
| 101 | +\| Quant \| c16 tok/s chart \| |
| 102 | +\|---\|---\| |
| 103 | +\| Q4\_K\_M \| \`################################################################ 655.6\` \| |
| 104 | +\| Q5\_K\_M \| \`############################################################## 638.6\` \| |
| 105 | +\| Q6\_K \| \`########################################################### 603.3\` \| |
| 106 | +\| Q8\_0 \| \`########################################################### 601.4\` \| |
| 107 | +\| Q3\_K\_M \| \`################################################ 493.0\` \| |
| 108 | +\| IQ4\_XS \| \`############################################### 476.0\` \| |
| 109 | + |
| 110 | +\### KLD Quality Chart |
| 111 | + |
| 112 | +KLD method: next-token top-64 approximate \`KL(P\_bf16 \|\| P\_quant)\` over 32 |
| 113 | +coding prompts, native llama.cpp \`/completion\`, \`n\_predict=1\`, \`temperature=-1\`, |
| 114 | +\`n\_probs=64\`, token-ID matching. Lower is better. |
| 115 | + |
| 116 | +\| Quant \| Mean KLD nats \| Mean KLD bits \| P95 nats \| Max nats \| Top-1 match \| |
| 117 | +\|---\|---:\|---:\|---:\|---:\|---:\| |
| 118 | +\| Q3\_K\_M \| 0.3620 \| 0.5223 \| 1.1077 \| 1.3730 \| 27/32 (84.4%) \| |
| 119 | +\| IQ4\_XS \| 0.1426 \| 0.2057 \| 0.3195 \| 0.6586 \| 27/32 (84.4%) \| |
| 120 | +\| Q4\_K\_M \| 0.0864 \| 0.1247 \| 0.2877 \| 0.4503 \| 29/32 (90.6%) \| |
| 121 | +\| Q5\_K\_M \| 0.0354 \| 0.0510 \| 0.0943 \| 0.2497 \| 30/32 (93.8%) \| |
| 122 | +\| Q6\_K \| 0.0165 \| 0.0238 \| 0.0513 \| 0.0586 \| 32/32 (100.0%) \| |
| 123 | +\| Q8\_0 \| 0.0108 \| 0.0156 \| 0.0440 \| 0.0590 \| 31/32 (96.9%) \| |
| 124 | + |
| 125 | +Mean KLD drift bar, lower is better: |
| 126 | + |
| 127 | +\| Quant \| Mean KLD chart \| |
| 128 | +\|---\|---\| |
| 129 | +\| Q8\_0 \| \`# 0.0108\` \| |
| 130 | +\| Q6\_K \| \`## 0.0165\` \| |
| 131 | +\| Q5\_K\_M \| \`#### 0.0354\` \| |
| 132 | +\| Q4\_K\_M \| \`######### 0.0864\` \| |
| 133 | +\| IQ4\_XS \| \`############## 0.1426\` \| |
| 134 | +\| Q3\_K\_M \| \`#################################### 0.3620\` \| |
| 135 | + |
| 136 | +Interpretation: |
| 137 | + |
| 138 | +\- Q4\_K\_M is the speed release target. |
| 139 | +\- Q3\_K\_M is valid for minimum memory, but has the largest KLD drift. |
| 140 | +\- IQ4\_XS is useful as a middle-footprint body and as the MTP graft base. |
| 141 | +\- Q5\_K\_M, Q6\_K, and Q8\_0 improve distribution fidelity but cost more VRAM. |
| 142 | + |
| 143 | +\### SWE-bench Agent Scores |
| 144 | + |
| 145 | +Full report: \`benchmarks/swebench-agent-evals.md\`. |
| 146 | + |
| 147 | +The full 100-task SWE-bench Verified run used Q6\_K with reasoning enabled, |
| 148 | +\`temperature=1.0\`, \`top\_p=1.0\`, and no explicit thinking budget: |
| 149 | + |
| 150 | +\| Profile \| Dataset slice \| Reasoning \| Resolved \| Score \| |
| 151 | +\|---\|---\|---\|---:\|---:\| |
| 152 | +\| Q6\_K GGUF \| \`SWE-bench/SWE-bench\_Verified\` test \`0:100\` \| on \| 69/100 \| 69.0% \| |
| 153 | + |
| 154 | +The multi-quant sweep used \`MariusHobbhahn/swe-bench-verified-mini\`, split |
| 155 | +\`test\`, 50 tasks: |
| 156 | + |
| 157 | +\| Quant/profile \| Resolved \| Score \| Completed \| Empty patch \| Errors \| |
| 158 | +\|---\|---:\|---:\|---:\|---:\|---:\| |
| 159 | +\| Q6\_K \| 33/50 \| 66.0% \| 48/50 \| 2 \| 0 \| |
| 160 | +\| Q3\_K\_M \| 32/50 \| 64.0% \| 47/50 \| 3 \| 0 \| |
| 161 | +\| Q5\_K\_M \| 32/50 \| 64.0% \| 49/50 \| 1 \| 0 \| |
| 162 | +\| Q4\_K\_M \| 31/50 \| 62.0% \| 48/50 \| 2 \| 0 \| |
| 163 | +\| IQ4\_XS \| 30/50 \| 60.0% \| 45/50 \| 5 \| 0 \| |
| 164 | +\| IQ4\_XS-MTP-graft-headQ6 \| 30/50 \| 60.0% \| 48/50 \| 2 \| 0 \| |
| 165 | +\| Q8\_0 \| 29/50 \| 58.0% \| 47/50 \| 3 \| 0 \| |
| 166 | + |
| 167 | +\## MTP Work |
| 168 | + |
| 169 | +\### Starting Problem |
| 170 | + |
| 171 | +The upstream Ornith config advertises one MTP/NextN layer, but the released HF |
| 172 | +weights and plain GGUF files do not contain a loadable MTP head. In practice, |
| 173 | +the plain model cannot simply be started with llama.cpp \`--spec-type draft-mtp\`. |
| 174 | +The work therefore split into two lines: |
| 175 | + |
| 176 | +1\. Make a loadable MTP draft path for experimentation. |
| 177 | +2\. Find a release artifact and serving policy that gives real single-user speed. |
| 178 | + |
| 179 | +\### Bootstrap Draft |
| 180 | + |
| 181 | +The first bootstrap created a minimal external MTP draft by cloning trunk layer |
| 182 | +39 into the expected MTP layout, adding the missing MTP projection/norm tensors, |
| 183 | +converting with \`convert\_hf\_to\_gguf.py --mtp\`, and quantizing the draft to |
| 184 | +IQ4\_XS. |
| 185 | + |
| 186 | +Result: |
| 187 | + |
| 188 | +\| Profile \| Loaded VRAM GiB \| c16 tok/s \| Acceptance \| Verdict \| |
| 189 | +\|---\|---:\|---:\|---:\|---\| |
| 190 | +\| IQ4\_XS target only \| 19.34 \| 476.0 \| n/a \| baseline \| |
| 191 | +\| Bootstrap IQ4\_XS MTP draft, \`n\_max=1\` \| 21.32 \| 162.7 \| 1.21% \| not recommended \| |
| 192 | + |
| 193 | +This proved the server plumbing and artifact layout but did not provide useful |
| 194 | +acceptance. |
| 195 | + |
| 196 | +\### Trained External Drafts |
| 197 | + |
| 198 | +The second line trained chain-corrected Qwen3.5 MoE MTP drafts from target |
| 199 | +hidden-state captures. The goal was to train a draft that predicts future tokens |
| 200 | +from the target hidden state and can be exported as a separate MTP GGUF. |
| 201 | + |
| 202 | +Best external draft family: |
| 203 | + |
| 204 | +\| Draft artifact \| Quant \| Loaded VRAM GiB \| c16/128 tok/s \| Acceptance \| Acceptance by position \| |
| 205 | +\|---\|---\|---:\|---:\|---:\|---\| |
| 206 | +\| \`...h2r9fw125...Q6\_K.gguf\` \| Q6\_K \| 22.44 \| 286.77 \| 67.22% \| \`(0.719, 0.621)\` \| |
| 207 | +\| \`...h2r9fw125...Q5\_K\_M.gguf\` \| Q5\_K\_M \| 22.16 \| 285.32 \| 66.61% \| \`(0.715, 0.613)\` \| |
| 208 | +\| \`...h2r9fw125...IQ4\_XS.gguf\` \| IQ4\_XS \| 21.53 \| 265.06 \| 59.20% \| \`(0.640, 0.541)\` \| |
| 209 | + |
| 210 | +The Q6\_K draft reached the requested development acceptance band, but the |
| 211 | +serving path remained slower than the target-only matched baseline: |
| 212 | + |
| 213 | +\| Profile \| c1 tok/s \| c4 tok/s \| c8 tok/s \| c16 tok/s \| |
| 214 | +\|---\|---:\|---:\|---:\|---:\| |
| 215 | +\| IQ4\_XS target only, no prompt cache \| 239.91 \| 431.95 \| 470.81 \| 595.89 \| |
| 216 | +\| IQ4\_XS + Q6\_K trained MTP draft, \`n\_max=2\`, per-head contexts \| 235.99 \| 245.28 \| 247.82 \| 288.67 \| |
| 217 | +\| IQ4\_XS + Q6\_K trained MTP draft, sequential verify \| 192.48 \| 165.63 \| 167.47 \| 169.24 \| |
| 218 | + |
| 219 | +Conclusion: the external draft quality became good enough for development, but |
| 220 | +the separate-draft runtime path was not a production speedup. |
| 221 | + |
| 222 | +\### Integrated IQ4\_XS-MTP Graft |
| 223 | + |
| 224 | +The release MTP path is the integrated graft: |
| 225 | + |
| 226 | +\`ornith-1.0-35b-IQ4\_XS-MTP-graft-headQ6.gguf\` |
| 227 | + |
| 228 | +This file was made by starting from the MIT-licensed |
| 229 | +\`wang-yang/Ornith-1.0-35B-MTP-GGUF\` Q6\_K graft and requantizing the body to |
| 230 | +IQ4\_XS while preserving the appended \`blk.40.\*\` MTP block at Q6\_K/F16/F32: |
| 231 | + |
| 232 | +\`\`\`bash |
| 233 | +llama-quantize --allow-requantize \ |
| 234 | + --tensor-type-file runs/tensor-types-mtp-blk40-preserve.txt \ |
| 235 | + Ornith-1.0-35B-Q6\_K-MTP.gguf \ |
| 236 | + ornith-1.0-35b-IQ4\_XS-MTP-graft-headQ6.gguf \ |
| 237 | + IQ4\_XS 24 |
| 238 | +\`\`\` |
| 239 | + |
| 240 | +Structural verification: |
| 241 | + |
| 242 | +\| Key \| Value \| |
| 243 | +\|---\|---\| |
| 244 | +\| \`general.architecture\` \| \`qwen35moe\` \| |
| 245 | +\| \`qwen35moe.block\_count\` \| \`41\` \| |
| 246 | +\| \`qwen35moe.nextn\_predict\_layers\` \| \`1\` \| |
| 247 | +\| MTP tensors \| appended \`blk.40.\*\` block with \`blk.40.nextn.\*\` tensors \| |
| 248 | +\| Body quant \| IQ4\_XS \| |
| 249 | +\| MTP block quant \| preserved Q6\_K/F16/F32 \| |
| 250 | +\| File size \| 19.6 GB decimal, about 0.69 GB over plain IQ4\_XS \| |
| 251 | + |
| 252 | +The integrated graft reproduced the high-acceptance behavior reported by the |
| 253 | +wang-yang reference and avoided the heavy external draft context. |
| 254 | + |
| 255 | +Low-concurrency results: |
| 256 | + |
| 257 | +\| Profile \| Shape \| tok/s \| Speedup \| Draft acceptance \| |
| 258 | +\|---\|---\|---:\|---:\|---:\| |
| 259 | +\| Integrated graft, target-only AR \| c1/128 \| 221.80 \| 1.00x \| n/a \| |
| 260 | +\| Integrated graft, \`draft-mtp n\_max=2\` \| c1/128 \| 279.36 \| 1.26x \| high \| |
| 261 | +\| Integrated graft, adaptive MTP throttle \| c1/128 \| 319.53 \| 1.44x \| \`(0.953, 0.865)\` \| |
| 262 | + |
| 263 | +Adaptive c1 server counters: |
| 264 | + |
| 265 | +\| Counter \| Value \| |
| 266 | +\|---\|---:\| |
| 267 | +\| Generated speculative tokens \| 339 \| |
| 268 | +\| Accepted speculative tokens \| 309 \| |
| 269 | +\| Mean accepted length \| 2.82 \| |
| 270 | +\| Acceptance by position \| \`(0.953, 0.865)\` \| |
| 271 | + |
| 272 | +Saturated c16 results: |
| 273 | + |
| 274 | +\| Profile \| Shape \| tok/s \| Ratio vs AR \| Notes \| |
| 275 | +\|---\|---\|---:\|---:\|---\| |
| 276 | +\| Target-only AR \| c16/64 \| 568.57 \| 1.00x \| matched control \| |
| 277 | +\| Always-on active MTP, \`n\_max=2\` \| c16/64 \| 323.12 \| 0.57x \| high acceptance but slower \| |
| 278 | +\| Adaptive MTP throttle, \`LLAMA\_SPEC\_MAX\_DRAFTING\_SLOTS=1\` \| c16/64 \| 564.48 \| 0.99x \| MTP disabled while saturated \| |
| 279 | + |
| 280 | +The final serving policy is therefore adaptive MTP: enable MTP for a single |
| 281 | +active request and disable drafting when the server is saturated. |
| 282 | + |
| 283 | +Threshold sweep: |
| 284 | + |
| 285 | +\| \`LLAMA\_SPEC\_MAX\_DRAFTING\_SLOTS\` \| c1 tok/s \| c2 tok/s \| c4 tok/s \| c8 tok/s \| c16 tok/s \| |
| 286 | +\|---:\|---:\|---:\|---:\|---:\|---:\| |
| 287 | +\| 1 \| 271.12 \| 316.46 \| 424.14 \| 564.24 \| 592.45 \| |
| 288 | +\| 2 \| 267.04 \| 313.25 \| 422.88 \| 558.17 \| 592.39 \| |
| 289 | +\| 4 \| 269.23 \| 313.06 \| 328.45 \| 547.56 \| 591.50 \| |
| 290 | + |
| 291 | +Recommended default: \`LLAMA\_SPEC\_MAX\_DRAFTING\_SLOTS=1\`. |
| 292 | + |
| 293 | +\### Dedicated MTP KLD and Sequence Evaluation |
| 294 | + |
| 295 | +See \[benchmarks/mtp-dedicated-kld-evaluation.md\](benchmarks/mtp-dedicated-kld-evaluation.md) |
| 296 | +for raw paths and full details. |
| 297 | + |
| 298 | +Next-token top-64 KLD over 32 coding prompts: |
| 299 | + |
| 300 | +\| Candidate \| Mean KLD nats \| Mean KLD bits \| P95 nats \| Max nats \| Top-1 match \| |
| 301 | +\|---\|---:\|---:\|---:\|---:\|---:\| |
| 302 | +\| Integrated IQ4\_XS-MTP graft, target-only \| 0.0731382442 \| 0.1055161823 \| 0.1593743642 \| 0.3348258511 \| 29/32 \| |
| 303 | +\| Integrated IQ4\_XS-MTP graft, active \`draft-mtp\` \| 0.0731382442 \| 0.1055161823 \| 0.1593743642 \| 0.3348258511 \| 29/32 \| |
| 304 | +\| Active \`draft-mtp\` vs target-only graft \| 0.0 \| 0.0 \| 0.0 \| 0.0 \| 32/32 \| |
| 305 | + |
| 306 | +The one-token API-visible distribution is clean: active \`draft-mtp\` returns the |
| 307 | +same next-token distribution as the graft running target-only. The integrated |
| 308 | +graft also measured lower drift than the prior plain IQ4\_XS capture |
| 309 | +(0.0731382442 vs 0.1425748206 mean KLD nats). |
| 310 | + |
| 311 | +The longer 8 prompt x 64 token single-user sequence probe exercised real native |
| 312 | +MTP drafting: |
| 313 | + |
| 314 | +\| Runtime \| Client aggregate tok/s \| Draft acceptance \| Exact 64-token sequences \| Token-position match \| |
| 315 | +\|---\|---:\|---:\|---:\|---:\| |
| 316 | +\| Target-only graft \| 172.57 \| n/a \| 8/8 baseline \| n/a \| |
| 317 | +\| Fast active \`draft-mtp\`, \`n\_max=2\` \| 233.81 \| 310/378 = 82.01%; \`(0.884, 0.747)\` by position \| 6/8 \| 478/512 = 93.36% \| |
| 318 | +\| \`LLAMA\_SPEC\_VERIFY\_SEQUENTIAL=1\` \| 164.34 \| 313/375 = 83.47%; \`(0.884, 0.772)\` by position \| 5/8 \| 455/512 = 88.87% \| |
| 319 | + |
| 320 | +This means the native graft remains an experimental low-concurrency speed |
| 321 | +profile, not a strict target-equivalent deterministic serving mode. Also, |
| 322 | +llama.cpp omitted candidate top-logprobs for most accepted speculative tokens |
| 323 | +in sequence responses: 500/512 positions were missing candidate top-logprobs in |
| 324 | +the fast active run. Comparable same-emitted-token positions had near-zero KLD |
| 325 | +(0.0011215739 mean nats, 0.0040239103 max), but full sequence KLD is not valid |
| 326 | +without runtime support for returning verifier logprobs on accepted draft tokens. |
| 327 | + |
| 328 | +\### Runtime Engineering Done |
| 329 | + |
| 330 | +The MTP work included artifact surgery and runtime work: |
| 331 | + |
| 332 | +\| Area \| Work completed \| Impact \| |
| 333 | +\|---\|---\|---\| |
| 334 | +\| Missing MTP tensors \| Built bootstrap MTP draft from trunk layer 39 and added required projection/norm tensors \| Made llama.cpp \`draft-mtp\` loadable for local experiments \| |
| 335 | +\| Hidden-state capture \| Added/used capture paths for target hidden rows and token positions \| Enabled trained external MTP draft experiments \| |
| 336 | +\| Chain-corrected training \| Trained h2/h1/h3 chain-corrected drafts and exported Q6\_K/Q5\_K\_M/IQ4\_XS GGUFs \| Raised external-draft acceptance from about 1% to about 67% \| |
| 337 | +\| Integrated graft requantization \| Requantized a Q6\_K integrated MTP graft to IQ4\_XS while preserving \`blk.40.\*\` MTP tensors \| Produced a 19.6 GB native MTP artifact \| |
| 338 | +\| Fast backend sampling \| Added a fast path for unambiguous backend top-k=1 draft sampling \| Reduced sampling overhead in the native graft path \| |
| 339 | +\| Qwen35MoE recurrent row index \| Patched recurrent-state writes and rollback snapshots to use runtime row-index tensors rather than graph-baked head offsets \| Fixed graph/recurrent mismatch across MTP head changes \| |
| 340 | +\| Per-head MTP contexts \| Added optional \`LLAMA\_MTP\_PER\_HEAD\_CONTEXT=1\` path for trained external drafts \| Improved graph reuse but did not make external drafts faster than target-only \| |
| 341 | +\| Sequential verification \| Added \`LLAMA\_SPEC\_VERIFY\_SEQUENTIAL=1\` correctness guard \| Made temperature-0 MTP output match target-only in deterministic probes \| |
| 342 | +\| Adaptive throttling \| Added/validated \`LLAMA\_SPEC\_MAX\_DRAFTING\_SLOTS=1\` policy \| Keeps c1 MTP speedup while avoiding saturated c16 regression \| |
| 343 | + |
| 344 | +Important correctness note: the external trained-draft path can diverge from |
| 345 | +target-only output without sequential verification. The sequential verifier |
| 346 | +matches target-only output but is slower, so it is a debug/correctness guard, not |
| 347 | +the production profile. The integrated graft's release value is low-concurrency |
| 348 | +speed with adaptive throttling. |
| 349 | + |
| 350 | +\### MTP Recommendation |
| 351 | + |
| 352 | +Use this single-GPU command for the integrated MTP profile: |
| 353 | + |
| 354 | +\`\`\`bash |
| 355 | +export CUDA\_VISIBLE\_DEVICES=0 |
| 356 | +export LLAMA\_SPEC\_MAX\_DRAFTING\_SLOTS=1 |
| 357 | +export LLAMA\_MTP\_FAST\_BACKEND\_SAMPLE=1 |
| 358 | +export LLAMA\_MTP\_DRAFT\_TOP\_K=1 |
| 359 | +export LLAMA\_MTP\_DRAFT\_TOP\_P=1 |
| 360 | +export LLAMA\_MTP\_DRAFT\_TEMP=1 |
| 361 | + |
| 362 | +llama-server \ |
| 363 | + -m ornith-1.0-35b-IQ4\_XS-MTP-graft-headQ6.gguf \ |
| 364 | + --alias Ornith-1.0-35B-GGUF-IQ4\_XS-MTP \ |
| 365 | + --host 127.0.0.1 --port 8002 \ |
| 366 | + -c 131072 -np 1 -b 4096 -ub 512 -ngl 99 \ |
| 367 | + --flash-attn on \ |
| 368 | + --reasoning off --reasoning-format deepseek \ |
| 369 | + --spec-type draft-mtp \ |
| 370 | + --spec-draft-ngl 99 \ |
| 371 | + --spec-draft-n-max 2 \ |
| 372 | + --spec-draft-n-min 0 \ |
| 373 | + --spec-draft-backend-sampling \ |
| 374 | + --cont-batching \ |
| 375 | + --metrics |
| 376 | +\`\`\` |
| 377 | + |
| 378 | +For multi-slot server traffic, keep \`LLAMA\_SPEC\_MAX\_DRAFTING\_SLOTS=1\` so MTP is |
| 379 | +used only when the server has a small active batch. For a single-user agentic |
| 380 | +workload, this is the preferred MTP profile. |
| 381 | + |
| 382 | +\## Pi-Agent Coding Smoke |
| 383 | + |
| 384 | +A Pi-agent smoke was run against the single-user integrated MTP profile with |
| 385 | +\`CTX\_SIZE=131072\`, \`PARALLEL=1\`, and model alias \`ornith-pi-agent-mtp\`. |
| 386 | + |
| 387 | +\| Test \| Result \| |
| 388 | +\|---\|---\| |
| 389 | +\| Calculator tool smoke \| pass \| |
| 390 | +\| Ledger parser/summary repo-edit task \| pass, 3/3 tests \| |
| 391 | +\| TTL cache repo-edit task \| pass, 3/3 tests \| |
| 392 | +\| Router dispatch repo-edit task \| pass, 4/4 tests \| |
| 393 | + |
| 394 | +Serving counters from that Pi-agent run: |
| 395 | + |
| 396 | +\| Metric \| Value \| |
| 397 | +\|---\|---:\| |
| 398 | +\| Prompt throughput \| 3460.27 tok/s \| |
| 399 | +\| Decode throughput \| 339.23 tok/s \| |
| 400 | +\| MTP generated draft tokens \| 1234 \| |
| 401 | +\| MTP accepted draft tokens \| 1114 \| |
| 402 | +\| Aggregate draft acceptance \| 90.3% \| |
| 403 | +\| Acceptance by position \| \`(0.929, 0.877)\` \| |
| 404 | +\| Mean accepted length \| 2.81 \| |
| 405 | +\| Peak observed loaded VRAM \| about 22.65 GiB \| |
| 406 | + |
| 407 | +\## Pi-Agent Quant Flappy Bird Test |
| 408 | + |
| 409 | +Each published serving profile was loaded as a single-user llama.cpp server and |
| 410 | +attached to Pi-agent with the same vanilla HTML/CSS/JS Flappy Bird prompt. The |
| 411 | +generated games are not hand-edited; Q3\_K\_M required three Pi-agent repair |
| 412 | +passes, and the other profiles passed on the first agent run. |
| 413 | + |
| 414 | +Full report: \`benchmarks/pi-agent-flappy-quant-test.md\`. |
| 415 | +Raw evidence: \`probes/pi-agent-flappy/\`. |
| 416 | + |
| 417 | +\| Quant/Profile \| Result \| Decode TPS on Pi-agent task \| |
| 418 | +\|---\|---:\|---:\| |
| 419 | +\| Q3\_K\_M \| pass after 3 Pi-agent repair passes \| 250.88 \| |
| 420 | +\| IQ4\_XS \| pass first run \| 242.44 \| |
| 421 | +\| Q4\_K\_M \| pass first run \| 249.59 \| |
| 422 | +\| Q5\_K\_M \| pass first run \| 239.67 \| |
| 423 | +\| Q6\_K \| pass first run \| 226.51 \| |
| 424 | +\| Q8\_0 \| pass first run \| 212.98 \| |
| 425 | +\| IQ4\_XS-MTP-graft-headQ6 \| pass first run \| 320.91 \| |
| 426 | + |
| 427 | +Interaction checks loaded every generated page, verified one canvas, pressed |
| 428 | +Space, captured an after-input screenshot, and observed zero page errors for all |
| 429 | +7 profiles. |
| 430 | + |
| 431 | +The MTP profile's cumulative server log for this agent workload reports 3,452 |
| 432 | +accepted draft tokens out of 3,882 generated draft tokens, 88.92% draft-token |
| 433 | +acceptance, mean acceptance length 2.78, and per-position acceptance |
| 434 | +\`(0.936, 0.843)\`. |
| 435 | + |
| 436 | +\## Evidence Index |
| 437 | + |
| 438 | +\- Quant benchmarks: \`benchmarks/llamacpp-quant-benchmarks.md\` |
| 439 | +\- Quant KLD probe: \`benchmarks/kld-quant-vs-bf16-top64.md\` |
| 440 | +\- Integrated MTP profile: \`benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md\` |
| 441 | +\- SWE-bench agent evaluations: \`benchmarks/swebench-agent-evals.md\` |
| 442 | +\- Pi-agent quant Flappy Bird test: \`benchmarks/pi-agent-flappy-quant-test.md\` |
| 443 | + with raw logs and metrics in \`probes/pi-agent-flappy/\` |
| 444 | +\- Trained MTP profile: \`benchmarks/llamacpp-iq4-xs-mtp-chaincorr-profile.md\` |
| 445 | +\- MTP profile catalog: \`configs/mtp\_profiles.yaml\` |
| 446 | +\- Quant artifact catalog: \`configs/quant\_artifacts.yaml\` |
| 447 | +\- Serving profile catalog: \`configs/serving\_profiles.yaml\` |
| 448 | +\- Runtime patches: \`patches/llamacpp-qwen35moe-mtp-recurrent-row-index.patch\`, |
| 449 | + \`patches/llamacpp-mtp-adaptive-serving-and-recurrent-rollback.patch\`, and |
| 450 | + \`patches/llamacpp-server-mtp-seqverify-and-perhead.patch\` |

README.mdCHANGED
Viewed

|     |     |
| --- | --- |
|  | @@ -16,270 +16,437 @@ tags: |
| 16 | \- tp1 |
| 17 | \-\-\- |
| 18 |  |
| 19 | -\# Ornith ~~-1.0-~~ 35B ~~— GGUF (llama.cpp, single-GPU \`tp=1\`)~~ |
| 20 |  |
| 21 | -~~Single-GPU~~ ~~llama.cpp~~ ~~GGUF~~ ~~package~~ for ~~\[~~ \`deepreinforce-ai/Ornith-1.0-35B\` ~~\](https://huggingface~~. ~~co/deepreinforce-ai/Ornith-1.0-35B).~~ |
| 22 | -The supported serving policy is \`tp=1\` — one model copy per GPU. Multi-GPU |
| 23 | -tensor-parallel serving is intentionally out of scope for this version. |
| 24 |  |
| 25 | -~~This~~ ~~release~~ ~~ships six body quants (Q3\_K\_M → Q8\_0) plus an \*\*integrated~~ |
| 26 | -IQ4\_XS-MTP graft\*\* that adds a native multi-token-prediction (MTP) draft head for |
| 27 | -low-concurrency speculative decode. Quant quality is measured against the upstream |
| 28 | -BF16 GGUF with a native llama.cpp next-token top-64 KL-divergence probe over 32 |
| 29 | -coding prompts. |
| 30 |  |
| 31 | -~~##~~ ~~TL;DR~~ ~~—~~ ~~pick an artifact~~ |
|  |  |
|  |  |
|  |  |
|  |  |
| 32 |  |
| 33 | \| Use case \| Recommended artifact \| Key numbers \| |
| 34 | \|---\|---\|---\| |
| 35 | \| Default serving speed \| \`ornith-1.0-35b-Q4\_K\_M.gguf\` \| 19.71 GiB on disk, 21.31 GiB loaded VRAM, 243.3 tok/s c1, 655.6 tok/s c16 \| |
| 36 | \| Lowest memory \| \`ornith-1.0-35b-Q3\_K\_M.gguf\` \| 15.61 GiB on disk, 17.27 GiB loaded VRAM, 240.5 tok/s c1, 493.0 tok/s c16 \| |
| 37 | \| Middle footprint \| \`ornith-1.0-35b-IQ4\_XS.gguf\` \| 17.64 GiB on disk, 19.34 GiB loaded VRAM, 0.1426 mean top-64 KLD nats \| |
| 38 | -\| ~~Highest fidelity~~/ ~~~~ footprint \| \`ornith-1.0-35b-Q6\_K.gguf\` \| 26.56 GiB on disk, 28.03 GiB loaded VRAM, 0.0165 mean top-64 KLD nats, 32/32 top-1 \| |
| 39 | -\| Native low-concurrency MTP \| \`ornith-1.0-35b-IQ4\_XS-MTP-graft-headQ6.gguf\` \| 19.6 GB decimal, ~~\*\*0.0731 mean top-64 KLD (29/32)\*\*,~~ c1/128 adaptive MTP 319.53 tok/s, acceptance \`(0.953, 0.865)\` \| |
| 40 | - |
| 41 | -~~>~~ ~~\*\*MTP~~ ~~graft~~ ~~note:\*\*~~ ~~active~~ ~~MTP~~ ~~is~~ ~~byte-for-byte~~ ~~identical~~ ~~to~~ ~~target-only~~ ~~at the~~ |
| 42 | -~~>~~ next-token ~~(API~~- ~~visible)~~ ~~level,~~ ~~and~~ ~~gives~~ ~~~1.3×~~ ~~single-stream~~ ~~decode~~. ~~It~~ ~~is~~ |
| 43 | -~~>~~ ~~\*\*not\*\*~~ ~~bit-exact~~ ~~to~~ ~~target~~- ~~only~~ ~~over~~ ~~long~~ ~~deterministic sequences — see~~ |
| 44 | -~~>~~ ~~\[MTP\](#multi~~- ~~token-prediction-mtp).~~ ~~Use~~ ~~target-only~~ ~~for~~ ~~strict~~ ~~reproduction,~~ |
| 45 | -~~>~~ ~~MTP~~ ~~for~~ ~~throughput~~ ~~on~~ ~~open-ended~~ ~~work.~~ |
| 46 | - |
| 47 | -~~##~~ ~~Quantization~~ ~~&~~ ~~fidelity~~ |
| 48 | - |
| 49 | -~~Fidelity is \`KL(P\_bf16 \|\| P\_candidate)\` over next-token top-64 distributions,~~ |
| 50 | -~~32 coding prompts, \`n\_predict=1, temperature=-1, n\_probs=64\`, keyed by token ID.~~ |
| 51 | -~~Lower~~ KLD ~~and~~ ~~higher~~ ~~greedy~~ ~~top-1 agreement are better.~~ |
| 52 | - |
| 53 | -~~###~~ ~~Primary~~ ~~fidelity~~ ~~table~~ |
| 54 | - |
| 55 | -~~\|~~ ~~Artifact~~ ~~\|~~ ~~Mean~~ ~~KLD~~ ~~(nats)~~ ~~\|~~ ~~Greedy~~ ~~top~~- ~~1 † \|~~ |
| 56 | -~~\|~~- ~~--\|---:\|---:\|~~ |
| 57 | -~~\|~~ ~~\`Q8\_0\`~~ ~~\|~~ ~~0.0108~~ ~~\|~~ ~~31/32~~ ~~(96.9%)~~ ~~\|~~ |
| 58 | -~~\|~~ ~~\`Q6\_K\`~~ ~~\|~~ ~~0.0165~~ ~~\|~~ ~~32/32~~ ~~(100.0%)~~ ~~\|~~ |
| 59 | -~~\|~~ ~~\`Q5\_K\_M\` \| 0~~. ~~0354 \| 30/32 (93.8%) \|~~ |
| 60 | -~~\| \*\*\`IQ4\_XS-MTP\` (graft)\*\* \| \*\*0.0731382442\*\* \| \*\*29/32 (90.6%)\*\* \|~~ |
| 61 | -~~\|~~ ~~\`Q4\_K\_M\`~~ ~~\|~~ ~~0.0864 \| 29/32 (90.6%) \|~~ |
| 62 | -~~\| \`IQ4\_XS\` \| 0.1426 \| 27/32 (84.4%) \|~~ |
| 63 | -~~\|~~ ~~\`Q3\_K\_M\`~~ ~~\|~~ ~~0.3620~~ ~~\|~~ ~~27/32 (84.4%) \|~~ |
| 64 | - |
| 65 | -~~!\[Next-token fidelity ladder: mean top-64 KLD vs BF16 for the six body quants plus the IQ4\_XS-MTP graft, lower is better.\](assets/02\_fidelity\_ladder.png)~~ |
| 66 | - |
| 67 | -~~\*Fidelity ladder — the integrated IQ4\_XS-MTP graft (purple) lands between Q5\_K\_M~~ |
| 68 | -~~and~~ ~~Q4\_K\_M,~~ ~~i.e.~~ ~~better~~ ~~next-token~~ ~~fidelity~~ ~~than~~ ~~its~~ ~~Q4\_K\_M~~ ~~neighbor.\*~~ |
| 69 | - |
| 70 | -~~>~~ ~~\*\*†~~ ~~Top-1~~ ~~vs~~ ~~mean~~ ~~KLD~~ ~~are~~ ~~different~~ ~~axes.\*\*~~ ~~\*Top-1\*~~ ~~counts~~ ~~how~~ ~~often~~ ~~the single~~ |
| 71 | -~~\> most-likely (argmax) token matches BF16's, over the 32 prompts; \*mean KLD\*~~ |
| 72 | -~~\> measures divergence across the full top~~- ~~64~~ ~~distribution.~~ ~~Two~~ ~~quants~~ ~~can share a~~ |
| 73 | -~~\> top-1 count yet differ a lot in KLD — e.g. \`Q3\_K\_M\` and \`IQ4\_XS\` are both~~ |
| 74 | -~~>~~ ~~27~~/ ~~32~~ ~~on~~ ~~top-1,~~ ~~but~~ ~~\`Q3\_K\_M\`'s~~ ~~distribution~~ ~~diverges ~2.5× more (0.3620 vs~~ |
| 75 | -~~\> 0.1426 nats). Neither value is a typo.~~ |
| 76 | - |
| 77 | -~~-~~ ~~The~~ ~~\*\*integrated~~ ~~IQ4\_XS-MTP~~ ~~graft~~ ~~is 48~~. ~~70~~% ~~lower mean KLD than the plain~~ |
| 78 | -~~IQ4\_XS~~ ~~body\*\*~~ ~~(0.0731382442~~ ~~vs~~ ~~0~~. ~~1425748206),~~ ~~and improves greedy top-1 from~~ |
| 79 | -~~27/32~~ ~~to~~ ~~29/32~~. |
| 80 | -~~-~~ ~~\*\*Active~~ \` ~~draft-mtp~~ \` ~~≡~~ ~~target-only~~ ~~at the next-token level:\*\* their top-64~~ |
| 81 | -~~next~~- ~~token~~ ~~distributions~~ ~~are~~ ~~identical~~ ~~(KLD 0~~.0 ~~,~~ ~~32/32) for the one-token~~ |
| 82 | -~~API-visible~~ ~~probe~~. |
| 83 | - |
| 84 | -## ~~#~~ ~~Body-quant distribution detail (secondary)~~ |
| 85 | - |
| 86 | -~~Full per~~- ~~quant~~ ~~distribution~~ ~~from~~ ~~the~~ ~~corrected~~ ~~top-64~~ ~~probe~~ ~~(same~~ ~~32~~ ~~prompts)~~. |
| 87 | -~~\`Mean~~ ~~/~~ ~~P50~~ ~~/ P95 / Max\` are explicit nats statistics of~~ the ~~per-prompt~~ ~~top-64~~ |
| 88 | -~~KLD;~~ ~~in~~ ~~this~~ ~~run~~ ~~the~~ ~~BF16~~ ~~top-64~~ ~~distribution~~ ~~captured~~ ~~mean~~ ~~probability~~ ~~mass~~ |
| 89 | -~~0.999965,~~ ~~so~~ ~~the~~ ~~top-64~~ ~~approximation~~ ~~captured~~ ~~essentially~~ ~~all~~ ~~next-token~~ ~~mass~~. |
| 90 | - |
| 91 | -~~\|~~ ~~Quant~~ ~~\| GGUF GiB \| KLD-probe VRAM GiB \| Mean nats \| P50 nats \| P95 nats \| Max nats \| Top-1 † \|~~ |
| 92 | -~~\|---\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|~~ |
| 93 | -~~\| Q3\_K\_M \| 15~~. ~~61~~ ~~\|~~ ~~16.32~~ ~~\|~~ ~~0.3620~~ ~~\|~~ ~~0.2548~~ ~~\|~~ ~~1.1077~~ ~~\|~~ ~~1~~. ~~3730~~ ~~\| 27/32 (84~~. ~~4%) \|~~ |
| 94 | -~~\|~~ ~~IQ4\_XS~~ ~~\|~~ ~~17.64~~ ~~\|~~ ~~19.34~~ ~~\|~~ ~~0.1426~~ ~~\|~~ ~~0.0868~~ ~~\|~~ ~~0.3195~~ ~~\|~~ ~~0.6586~~ ~~\|~~ ~~27/32 (84~~. ~~4%) \|~~ |
| 95 | -~~\| Q4\_K\_M \| 19~~. ~~71~~ ~~\|~~ ~~20.35~~ ~~\|~~ ~~0.0864~~ ~~\|~~ ~~0.0379~~ ~~\|~~ ~~0.2877~~ ~~\|~~ ~~0~~. ~~4503 \| 29/32 (90.6%) \|~~ |
| 96 | -~~\| Q5\_K\_M \| 23~~. ~~03~~ ~~\|~~ ~~23.61~~ ~~\|~~ ~~0.0354~~ ~~\|~~ ~~0.0235~~ ~~\| 0~~. ~~0943 \| 0.2497 \| 30/32 (93.8%) \|~~ |
| 97 | -~~\| Q6\_K \| 26.56 \| 27.07 \| 0.0165 \| 0.0092 \| 0.0513 \| 0.0586 \| 32/32 (100.0%) \|~~ |
| 98 | -~~\|~~ ~~Q8\_0~~ ~~\|~~ ~~34~~. ~~37 \| 34.77 \| 0.0108 \| 0.0052 \| 0.0440 \| 0.0590 \| 31/32~~( ~~96~~. ~~9%~~) ~~\|~~ |
| 99 | - |
| 100 | -~~The IQ4\_XS~~- ~~MTP~~ ~~graft~~ ~~uses~~ ~~the~~ ~~same probe on a different breakpoint context; its~~ |
| 101 | -~~distribution detail (\`P50 0.0492811974~~/ ~~P95 0~~. ~~1593743642~~/ ~~Max 0~~. ~~3348258511\`~~) ~~is~~ |
| 102 | -~~in~~ ~~the~~ \[ ~~MTP~~\]( ~~#multi~~- ~~token~~- ~~prediction-mtp~~) ~~section~~. ~~Source:~~ |
| 103 | -~~\[benchmarks~~/ ~~kld-quant-vs-bf16-top64.md\](benchmarks/kld-quant-vs-bf16-top64.md).~~ |
| 104 | - |
| 105 | -~~##~~ ~~Serving~~ ~~performance~~ |
| 106 | - |
| 107 | -~~Aggregate decode throughput and p95 time~~- ~~to~~- ~~first~~- ~~token across concurrency,~~ |
| 108 | -~~llama.cpp~~ ~~\`tp=1\`,~~ ~~from~~ ~~\`scripts/bench\_openai.py~~ ~~--stream~~ ~~--max-tokens 256\`~~( ~~0~~ |
| 109 | -~~failed~~ ~~requests~~ ~~per~~ ~~row)~~. ~~Q4\_K\_M is the speed pick; Q5\_K\_M nearly ties it at c16~~. |
| 110 | - |
| 111 | -~~\|~~ ~~Quant~~ ~~\|~~ ~~c1 tok~~/ ~~s \| c1 p95 TTFT ms \| c4 tok~~/ ~~s \| c4 p95 TTFT ms \| c8 tok/s \| c8 p95 TTFT ms \| c16 tok/s \| c16 p95 TTFT ms \|~~ |
| 112 | -~~\|~~\-\-\- ~~\|~~- ~~--:\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|~~ |
| 113 | -~~\|~~ ~~Q3\_K\_M~~ ~~\|~~ ~~240~~. ~~5 \| 77~~. ~~9 \| 422.0 \| 170.7 \| 464.4 \| 344.9 \| 493.0 \| 493.7 \|~~ |
| 114 | -~~\|~~ ~~IQ4\_XS~~ ~~\|~~ ~~234~~. ~~1 \| 75~~. ~~1 \| 297~~. ~~7 \| 159.7 \| 411.5 \| 330.2 \| 476.0 \| 541.5 \|~~ |
| 115 | -~~\|~~ ~~Q4\_K\_M~~ ~~\|~~ ~~243~~. ~~3 \| 76~~. ~~3 \| 458.3 \| 192.3 \| 615.0 \| 361.8 \| 655.6 \| 650.0 \|~~ |
| 116 | -~~\|~~ ~~Q5\_K\_M~~ ~~\|~~ ~~236~~. ~~7 \| 75~~. ~~1 \| 311~~. ~~0 \| 198.8 \| 439.0 \| 383.6 \| 638.6 \| 620.4 \|~~ |
| 117 | -~~\|~~ ~~Q6\_K~~ ~~\|~~ ~~225~~. ~~9 \| 76~~. ~~8 \| 295.8 \| 194.2 \| 409.6 \| 394.8 \| 603.3 \| 657.4 \|~~ |
| 118 | -~~\|~~ ~~Q8\_0~~ ~~\|~~ ~~208~~. ~~5 \| 76~~. ~~9 \| 281~~. ~~5 \| 190.8 \| 405.8 \| 389.1 \| 601.4 \| 725.8 \|~~ |
| 119 | - |
| 120 | -~~!\[Aggregate~~ ~~decode~~ ~~throughput~~ ~~vs concurrency (c1–c16) for the six quants; Q4\_K\_M leads at c16~~.\]( ~~assets~~/ ~~03\_throughput\_tps~~. ~~png~~) |
| 121 | - |
| 122 | -~~!\[p95~~ ~~time~~- ~~to~~- ~~first-token vs concurrency (c1–c16) for the six quants; lower is better~~.\]( ~~assets~~/ ~~04\_ttft\_p95~~. ~~png~~) |
| 123 | - |
| 124 | -~~>~~ ~~\*\*Profile~~ ~~note:\*\* the table above is a \*\*short~~- ~~context\*\* profile — \`CTX\_SIZE=8192~~ |
| 125 | -~~\> PARALLEL=16\` exposes \`n\_ctx = 512\`/slot, with ~23~~- ~~token~~ ~~prompts~~ ~~and 256-token~~ |
| 126 | -~~>~~ ~~generations.~~ ~~Source:~~\[ ~~benchmarks~~/ ~~llamacpp~~- ~~quant~~- ~~benchmarks~~.md\]( ~~benchmarks~~/ ~~llamacpp~~- ~~quant~~- ~~benchmarks~~.md). |
| 127 | -~~>~~ ~~For~~ ~~how~~ ~~prefill latency scales with prompt length, see long~~- ~~context TTFT below~~. |
| 128 | - |
| 129 | -~~###~~ ~~Long-context~~ ~~TTFT~~ ( ~~single stream~~) |
| 130 | - |
| 131 | -~~p95~~ ~~time-to-first-token~~ ~~vs~~ ~~prompt~~/ ~~context length, single GPU \`tp=1\`, single stream~~ |
| 132 | -~~(\`CTX\_SIZE=131072~~ ~~PARALLEL=1\`, \`n\_ctx~~/ ~~slot=131072\`,~~ ~~exact~~ ~~prompt token counts, 140~~ |
| 133 | -~~rows/quant,~~ 0 failures ~~).~~ |
| 134 | - |
| 135 | -~~\|~~ ~~Context~~ ~~tokens~~ ~~\|~~ ~~Q4\_K\_M~~ ~~p50/p95~~ ~~ms~~ ~~\|~~ ~~Q4\_K\_M~~ ~~tok/s~~ ~~\|~~ ~~IQ4\_XS p50/p95 ms \| IQ4\_XS tok/s \| MTP-graft p50/p95 ms \| MTP-graft tok/s \|~~ |
| 136 | -~~\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|~~ |
| 137 | -~~\|~~ ~~512 \| 91.8~~/ ~~94~~. ~~1~~ ~~\|~~ ~~188.3~~ ~~\|~~ ~~87.1~~ ~~/ 88.7 \| 177.4 \| 86.0 / 87.8 \| 184.0 \|~~ |
| 138 | -~~\|~~ ~~1024~~ ~~\|~~ ~~169.6~~ ~~/~~ ~~172.2~~ ~~\|~~ ~~145~~. ~~8 \| 159.9 / 161.6 \| 139.9 \| 157.6 / 159.0 \| 155.9 \|~~ |
| 139 | -~~\|~~ ~~2048~~ ~~\|~~ ~~341.8~~ ~~/~~ ~~346.1~~ ~~\|~~ ~~97.9~~ ~~\|~~ ~~316.1~~ ~~/~~ ~~318~~. ~~7 \| 112~~. ~~0 \| 307~~. ~~6~~ ~~/ 308.9 \| 113.9 \|~~ |
| 140 | -~~\|~~ ~~4096~~ ~~\|~~ ~~699.2~~ ~~/~~ ~~702.2~~ ~~\|~~ ~~67.2~~ ~~\|~~ ~~647.0~~ ~~/~~ ~~653.4~~ ~~\| 70.7 \| 625.7 / 631.3 \| 69.0 \|~~ |
| 141 | -~~\|~~ ~~8192~~ ~~\| 1447~~. ~~5 / 1458.0 \| 33.7 \| 1342.9 / 1348.5 \| 39.8 \| 1297.4 / 1304.0 \| 40.9 \|~~ |
| 142 | -~~\| 16384 \| 3009.7 / 3030.4 \| 19.5 \| 2793.2 / 2804.0 \| 20.9 \| 2709.8 / 2737.3 \| 21.5 \|~~ |
| 143 | -~~\|~~ ~~32768~~ ~~\|~~ ~~6302.4 / 6313.0 \| 9.7 \| 5829.9 / 5853.1 \| 10.5 \| 5673.5 / 5696.6 \| 10.7 \|~~ |
| 144 | - |
| 145 | -~~!\[Long~~- ~~context~~ ~~p95~~ ~~TTFT~~ ~~vs~~ ~~prompt~~ ~~length~~ ~~(512–32768~~ ~~tokens) for~~ Q4\_K\_M ~~,~~ ~~IQ4\_XS, and the MTP graft; single stream,~~ tp=1. ~~\](assets/07\_longctx\_ttft.png)~~ |
| 146 | - |
| 147 | -~~p95~~ ~~TTFT~~ ~~rises~~ with ~~prompt~~ ~~length~~ ~~(prefill~~ ~~cost~~) ~~to~~ ~~~6~~.3 ~~s~~ ~~at~~ ~~32k~~ ~~tokens~~ ~~for~~ |
| 148 | -~~Q4\_K\_M;~~ ~~the~~ ~~IQ4\_XS~~ ~~body~~ and ~~IQ4\_XS-MTP~~ ~~graft~~ ~~prefill~~ ~~slightly~~ ~~faster~~ ~~at~~ ~~every~~ |
| 149 | -~~length~~. ~~Decode~~ ~~throughput~~ ~~falls~~ ~~from~~ ~~~180–190~~ ~~tok/s~~ ~~at~~ ~~512~~ ~~tokens~~ ~~to~~ ~~~10~~ ~~tok/s~~ ~~at~~ |
| 150 | -~~32k~~ as the ~~KV~~ ~~cache~~ ~~grows~~. ~~Source:~~ ~~\[benchmarks/llamacpp-longctx-ttft~~. ~~md\](benchmarks/llamacpp~~- ~~longctx~~- ~~ttft~~. ~~md)~~. |
| 151 | - |
| 152 | -~~\## Multi~~- ~~token~~ ~~prediction~~ ~~(MTP)~~ |
| 153 | - |
| 154 | -~~The~~ ~~integrated~~ \` ~~IQ4\_XS~~- ~~MTP~~- ~~graft-headQ6~~ \` ~~artifact carries a native MTP draft head~~ |
| 155 | -~~on~~ ~~the~~ ~~IQ4\_XS~~ ~~body. The release recommendation is \*\*adaptive\*\*~~: ~~use MTP for~~ |
| 156 | -~~low~~- ~~concurrency~~ ~~/~~ ~~single-user~~ ~~requests and keep \`LLAMA\_SPEC\_MAX\_DRAFTING\_SLOTS=1\`~~ |
| 157 | -so saturated batches fall back to target-only throughput. The MTP catalog |
| 158 | -validates the published MTP profiles. |
| 159 | - |
| 160 | -\### Next-token quality |
| 161 | - |
| 162 | -Active MTP \*\*does not change\*\* the one-token, API-visible next-token |
| 163 | -distribution. Mean BF16 KLD is \*\*0.0731382442 nats\*\* over the 32-prompt top-64 |
| 164 | -probe (P50 0.0492811974 / P95 0.1593743642 / Max 0.3348258511), top-1 29/32. The |
| 165 | -active \`draft-mtp\` vs target-only graft next-token KLD is \*\*0.0 (32/32 identical)\*\*. |
| 166 | - |
| 167 | -\### Sequence-level behavior |
| 168 | - |
| 169 | -A deterministic \*\*8 prompt × 64 token\*\* probe (\`n\_predict=64, temperature=-1, |
| 170 | -n\_probs=64\`) compares the same GGUF running active native \`draft-mtp\` against |
| 171 | -target-only. |
| 172 | - |
| 173 | -\| Runtime \| Client agg tok/s \| Server decode tok/s \| Draft acceptance \| Exact 64-tok seqs \| Token-position match \| |
| 174 | -\|---\|---:\|---\|---\|---:\|---\| |
| 175 | -\| Target-only graft \| 172.57 \| ~210 (per-request timing) \| n/a \| 8/8 baseline \| n/a \| |
| 176 | -\| Fast active \`draft-mtp\`, \`n\_max=2\` \| 233.81 \| 325.70 (/metrics) \| 310/378 = 82.01%; per-position \`(0.884, 0.747)\` \| 6/8 \| 478/512 = 93.36% \| |
| 177 | -\| \`LLAMA\_SPEC\_VERIFY\_SEQUENTIAL=1\` \| 164.34 \| 204.88 (/metrics) \| 313/375 = 83.47%; per-position \`(0.884, 0.772)\` \| 5/8 \| 455/512 = 88.87% \| |
| 178 | - |
| 179 | -!\[MTP single-stream decode speedup on the deterministic 8x64 probe: target-only vs active MTP graft, client and server decode.\](assets/06\_mtp\_tps.png) |
| 180 | - |
| 181 | -\- Fast active MTP gives \*\*~1.35× client throughput\*\* (233.81 vs 172.57) and |
| 182 | - server decode 325.70 vs ~210, with \*\*82.01%\*\* draft acceptance. |
| 183 | -\- It matches \*\*6/8\*\* sequences exactly and \*\*478/512 = 93.36%\*\* of token |
| 184 | - positions; the two non-exact sequences first diverge at positions \*\*60\*\* and |
| 185 | - \*\*34\*\*. |
| 186 | -\- The \`LLAMA\_SPEC\_VERIFY\_SEQUENTIAL=1\` variant is both \*\*slower\*\* (164.34 / |
| 187 | - 204.88) \*\*and\*\* matches less (5/8, 88.87%; divergences at 25, 43, 60) — i.e. the |
| 188 | - fast verifier is here both faster and more target-matching. |
| 189 | - |
| 190 | -\> \*\*Note:\*\* MTP self-speculation is \*\*not bit-exact to target-only over long |
| 191 | -\> deterministic generations\*\*. For workloads that require exact target |
| 192 | -\> reproduction, run target-only; for throughput on open-ended work, MTP wins. |
| 193 | -\> MTP is a single-user / low-concurrency win — it is not faster on saturated |
| 194 | -\> batches, which is why the adaptive throttle falls back to target-only. |
| 195 | - |
| 196 | -\### Sequence logprob comparability (API limitation) |
| 197 |  |
| 198 | -~~llama.cpp omits candidate~~ \` ~~top\_logprobs~~ \` ~~for most accepted speculative tokens, so~~ |
| 199 | -~~a~~ ~~naive~~ ~~"sequence KLD" looks~~\*\* ~~enormous\*\* — that is an \*\*API/logprob~~- ~~availability~~ |
| 200 | -~~artifact, not model quality\*\*. On the comparable subset where the emitted token~~ |
| 201 | -also matches, KLD is \*\*≈ 0.001 nats\*\* (near zero): |
| 202 |  |
| 203 | -~~\|~~ ~~Runtime~~ ~~\|~~ ~~Paired positions \| Comparable top-logprobs \| Missing \| Comparable same-token mean KLD (nats) \| Max (nats) \|~~ |
| 204 | -\|---\|---:\|---:\|---:\|---:\|---:\| |
| 205 | -\| Fast active \`draft-mtp\` vs target-only \| 512 \| 12 \| 500 \| 0.0011215739 \| 0.0040239103 \| |
| 206 | -\| Sequential verifier vs target-only \| 512 \| 10 \| 502 \| 0.0012293172 \| 0.0040239103 \| |
| 207 | - |
| 208 | -Full sequence KLD cannot be computed without changing the runtime to return |
| 209 | -target verifier logprobs for accepted speculative tokens. |
| 210 |  |
| 211 | -~~\### Draft~~- ~~head~~ ~~distillation~~ ~~provenance~~ |
|  |  |
|  |  |
|  |  |
|  |  |
| 212 |  |
| 213 | -~~The~~ ~~meaningful~~ ~~KL-style metric for a standalone MTP draft head is \*\*teacher KL~~ |
| 214 | -against cached target hidden states\*\*, not standard base-model next-token KLD. |
| 215 | -The draft head was trained and \*\*measurably improved\*\* across checkpoints — |
| 216 | -teacher KL fell 1.92 → 1.50 → 0.76 and teacher argmax top-1 rose 0.54 → 0.69 → |
| 217 | -0.89. |
| 218 |  |
| 219 | -~~\|~~ ~~Draft~~ ~~checkpoint~~ ~~\|~~ ~~Tokens~~ ~~\|~~ ~~Windows~~ ~~\| Teacher KL \| Teacher argmax top~~- ~~1~~ ~~\|~~ ~~Teacher~~ ~~argmax~~ ~~top-5~~ ~~\| Gold top-1 \|~~ |
| 220 | -~~\|~~- ~~--\|---~~: ~~\|~~- ~~--:\|---:\|---:\|---:\|---:\|~~ |
| 221 | -~~\| \`mtp~~- ~~distill-kl-step500\`~~ ~~\|~~ ~~22~~, ~~637~~ ~~\|~~ ~~128~~ ~~\|~~ ~~1.9222025748~~ ~~\|~~ ~~0~~. ~~5360250917 \|~~ 0. ~~7881786456 \| 0.4425939833 \|~~ |
| 222 | -~~\|~~ \` ~~snapshots-iq4-live-accepted-a/step-1000~~ \` ~~\|~~ ~~58,754~~ ~~\|~~ ~~413~~ ~~\|~~ ~~1~~. ~~5038724942~~ ~~\|~~ ~~0~~. ~~6876808388~~ ~~\|~~ ~~0~~. ~~8368451510~~ ~~\|~~ ~~0~~. ~~6764816013~~ ~~\|~~ |
| 223 | -\| \`snapshots-iq4-live-allrows-b/step-1000\` \| 32,264 \| 2,048 \| 0.7591610373 \| 0.8908070915 \| 0.9389722291 \| 0.8076183982 \| |
| 224 |  |
| 225 | -~~!\[MTP~~ ~~draft-head~~ ~~distillation progress: teacher KL falls (left) while top-1 agreement rises (right) across three checkpoints.\](assets/05\_mtp\_draft\_head\_distill.png)~~ |
| 226 |  |
| 227 | -~~\*Teacher~~ ~~KL~~ ~~is~~ ~~measured~~ ~~against cached target hidden states (not base next-token~~ |
| 228 | -KLD). These training artifacts live under \`artifacts/mtp/\` locally and are |
| 229 | -referenced by name as provenance — they are not committed in this HF repo.\* |
| 230 |  |
| 231 | -~~\## Run with llama.cpp~~ |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
| 232 |  |
| 233 | -~~Serve~~ ~~a~~ ~~body~~ ~~quant~~ ~~on~~ ~~GPU0~~ ~~(\`tp=1\`)~~: |
| 234 |  |
| 235 | \`\`\`bash |
| 236 | -~~QUANT=Q4\_K\_M~~ ~~PORT=8000~~ ~~CTX\_SIZE=8192~~ ~~PARALLEL=16 REASONING=off \~~ |
| 237 | -~~~~ scripts/ ~~serve\_llamacpp\_gpu0~~. ~~sh~~ |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
| 238 | \`\`\` |
| 239 |  |
| 240 | -~~Serve~~ ~~the~~ ~~integrated~~ ~~MTP~~ ~~graft~~ ~~for~~ ~~low-concurrency~~ ~~speculative~~ ~~decode~~ ~~(keep~~ |
| 241 | -adaptive throttling so saturated batches fall back to target-only): |
| 242 |  |
| 243 | \`\`\`bash |
| 244 | -~~QUANT~~ = ~~IQ4\_XS-MTP-graft-headQ6~~ ~~PORT=8000 CTX\_SIZE=8192 PARALLEL=1 \~~ |
| 245 | -~~CACHE\_RAM~~ =0 ~~REASONING=off LLAMA\_SPEC\_MAX\_DRAFTING\_SLOTS=1 \~~ |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
| 246 | scripts/serve\_llamacpp\_gpu0.sh |
| 247 | \`\`\` |
| 248 |  |
| 249 | -~~\`REASONING=off\`~~ ~~is~~ ~~the~~ ~~default~~ ~~because~~ ~~the~~ ~~model~~ ~~otherwise spends simple coding~~ |
| 250 | -~~prompts in \`reasoning\_content\` before producing final \`content\`. See~~ |
| 251 | -~~\[benchmarks/llamacpp-q4-reasoning-off-fix.md\](benchmarks/llamacpp-q4-reasoning-off-fix.md).~~ |
| 252 | -~~Serving~~ / ~~quant / MTP profile catalogs:~~ |
| 253 | -~~\[configs~~/ ~~serving\_profiles.yaml\](configs~~/ ~~serving\_profiles.yaml),~~ |
| 254 | -~~\[configs/quant\_artifacts~~. ~~yaml\](configs/quant\_artifacts.yaml),~~ |
| 255 | -~~\[configs/mtp\_profiles.yaml\](configs/mtp\_profiles.yaml).~~ |
| 256 | - |
| 257 | -~~##~~ ~~Benchmark environment~~ |
| 258 | - |
| 259 | -~~\*\*Hardware:\*\* NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 97,887 MiB~~ |
| 260 | -~~VRAM,~~ ~~single~~ ~~GPU~~ ~~(\`tp=1\`).~~ ~~Driver~~ ~~580.159.03,~~ ~~CUDA~~ ~~runtime~~ ~~reported by driver 13.0.~~ |
| 261 | - |
| 262 | -~~\*\*Dedicated MTP KLD eval config (2026-06-28):\*\* local llama.cpp CUDA server,~~ |
| 263 | -~~single~~ ~~\*\*RTX~~ ~~PRO 6000 Blackwell 96GB\*\*, \`tp=1\`, \`CTX\_SIZE=8192\`, \`PARALLEL=1\`,~~ |
| 264 | -~~\`CACHE\_RAM=0\`,~~ ~~\`REASONING=off\`.~~ |
| 265 | - |
| 266 | -\- ~~\*\*llama.cpp build~~/ ~~commit:\*\* \`050ee92d04c2e1f639025786dea701c70e7d4204\` (pinned for~~ |
| 267 | -~~the long-context TTFT run; earlier KLD/throughput sweeps were not separately pinned).~~ |
| 268 | -~~\- \*\*Reasoning mode:\*\* all benchmarks in this card ran with \`REASONING=off\`.~~ |
| 269 | - |
| 270 | -~~\## Provenance & reproducibility~~ |
| 271 | - |
| 272 | -\- ~~\*\*Figures:\*\* regenerate with \[\`~~ scripts/ ~~make\_charts~~.py ~~\`\](scripts/make\_charts.py)~~ |
| 273 | -~~→~~ ~~\`assets~~/ ~~02\_fidelity\_ladder.png\`, \`assets~~/ ~~03\_throughput\_tps~~. ~~png\`,~~ |
| 274 | -~~\`assets/04\_ttft\_p95.png\`,~~ ~~\`assets~~/ ~~05\_mtp\_draft\_head\_distill~~. ~~png\`,~~ |
| 275 | -~~\`assets/06\_mtp\_tps.png\`.~~ |
| 276 | -- ~~\*\*Serving throughput~~/ ~~TTFT:\*\* \[benchmarks~~/ ~~llamacpp-quant-benchmarks~~. ~~md\](benchmarks/llamacpp-quant-benchmarks.md).~~ |
| 277 | -- ~~\*\*Body~~- ~~quant~~ ~~fidelity:\*\* \[benchmarks~~/ ~~kld-quant-vs-bf16-top64.md\](benchmarks~~/ ~~kld-quant-vs-bf16-top64~~. ~~md).~~ |
| 278 | -\- ~~\*\*Dedicated MTP KLD~~/ ~~sequence~~ eval ~~(2026-06-28):\*\*~~ |
| 279 | -~~\[benchmarks/mtp~~- ~~kld~~- ~~eval-2026-06-28.md\](benchmarks~~/ ~~mtp~~- ~~kld~~- ~~eval-2026-06-28~~. ~~md)~~ |
| 280 | -~~and \[benchmarks/mtp-dedicated-kld-evaluation.md\](benchmarks/mtp-dedicated-kld-evaluation.md),~~ |
| 281 | -~~with raw evidence under \[benchmarks/raw/\](benchmarks/raw/). Includes the~~ |
| 282 | -~~sequence-level~~ ~~probe~~ ~~and~~ ~~its~~ ~~logprob-availability~~ ~~limitation~~ ~~(full~~ ~~sequence~~ ~~KLD~~ |
| 283 | -~~is~~ ~~not~~ ~~computable~~ ~~from~~ ~~the~~ ~~current~~ ~~llama.cpp~~ ~~API response)~~. |
| 284 | -~~\- \*\*Adaptive MTP serving profile:\*\* \[benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md\](benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md).~~ |
| 285 | -~~-~~ ~~\*\*Long-context~~ ~~TTFT:\*\*~~ ~~\[benchmarks/llamacpp-longctx-ttft~~. ~~md\](benchmarks/llamacpp-longctx-ttft.md).~~ |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

|     |     |
| --- | --- |
|  |  |
| 16 | \- tp1 |
| 17 | \-\-\- |
| 18 |  |
| 19 | +\# Ornith35B Lab |
| 20 |  |
| 21 | +Thisworkspacetracksoptimizationwork for \`deepreinforce-ai/Ornith-1.0-35B\`. |
|  |  |
|  |  |
| 22 |  |
| 23 | +##ReleaseSummary |
|  |  |
|  |  |
|  |  |
|  |  |
| 24 |  |
| 25 | +Fullreleasewriteup:\[MODEL\_RELEASE\_REPORT.md\](MODEL\_RELEASE\_REPORT.md). |
| 26 | + |
| 27 | +This release is a single-GPU llama.cpp GGUF package for Ornith-1.0-35B. The |
| 28 | +supported serving policy is \`tp=1\`, one model copy per GPU. Multi-GPU tensor |
| 29 | +parallel serving is intentionally out of scope for this version. |
| 30 |  |
| 31 | \| Use case \| Recommended artifact \| Key numbers \| |
| 32 | \|---\|---\|---\| |
| 33 | \| Default serving speed \| \`ornith-1.0-35b-Q4\_K\_M.gguf\` \| 19.71 GiB on disk, 21.31 GiB loaded VRAM, 243.3 tok/s c1, 655.6 tok/s c16 \| |
| 34 | \| Lowest memory \| \`ornith-1.0-35b-Q3\_K\_M.gguf\` \| 15.61 GiB on disk, 17.27 GiB loaded VRAM, 240.5 tok/s c1, 493.0 tok/s c16 \| |
| 35 | \| Middle footprint \| \`ornith-1.0-35b-IQ4\_XS.gguf\` \| 17.64 GiB on disk, 19.34 GiB loaded VRAM, 0.1426 mean top-64 KLD nats \| |
| 36 | +\| Quality/footprint \| \`ornith-1.0-35b-Q6\_K.gguf\` \| 26.56 GiB on disk, 28.03 GiB loaded VRAM, 0.0165 mean top-64 KLD nats, 32/32 top-1 match \| |
| 37 | +\| Native low-concurrency MTP \| \`ornith-1.0-35b-IQ4\_XS-MTP-graft-headQ6.gguf\` \| 19.6 GB decimal, c1/128 adaptive MTP 319.53 tok/s, acceptance \`(0.953, 0.865)\` \| |
| 38 | + |
| 39 | +QuantqualitywasmeasuredagainsttheupstreamBF16GGUFwithanative |
| 40 | +llama.cpp next-token top-64KLDprobeover32codingprompts. MTPwork includes |
| 41 | +abootstrappedexternaldraft,trained chain-correcteddraftartifacts,a |
| 42 | +requantizedintegrated IQ4\_XS-MTPgraft,andpatchedllama.cppruntime support |
| 43 | +forrecurrentrow-indexhandling,fastbackendsampling, sequential verification, |
| 44 | +per-head contexts, and adaptive MTP throttling. The release MTP recommendation |
| 45 | +isadaptive:useMTP for low-concurrency/single-user requests and keep |
| 46 | +\`LLAMA\_SPEC\_MAX\_DRAFTING\_SLOTS=1\` so saturated batches fall back to target-only |
| 47 | +throughput. |
| 48 | + |
| 49 | +DedicatedMTP KLD/sequenceevaluationisavailableat |
| 50 | +\[benchmarks/mtp-dedicated-kld-evaluation.md\](benchmarks/mtp-dedicated-kld-evaluation.md). |
| 51 | +TheintegratedIQ4\_XS-MTPgraft measured 0.073138 mean top-64 |
| 52 | +\`KL(P\_bf16 \|\| P\_candidate)\` nats over 32 prompts. In an 8 prompt x 64 token |
| 53 | +single-usersequenceprobe,fastnative\`draft-mtp\`reached82.01%draft-token |
| 54 | +acceptance and 233.81 client tok/s vs 172.57 target-only, but matched only |
| 55 | +478/512generatedtokenpositions,sotheactiveMTP path remains an |
| 56 | +experimentallow-concurrencyspeedprofileratherthanastrict target-equivalent |
| 57 | +servingmode. |
| 58 | + |
| 59 | +##SWE-benchAgentScores |
| 60 | + |
| 61 | +FullreportandrawscoreJSON: |
| 62 | +\[benchmarks/swebench-agent-evals.md\](benchmarks/swebench-agent-evals.md). |
| 63 | + |
| 64 | +Full SWE-bench Verified 100-task slice, reasoning mode enabled: |
| 65 | + |
| 66 | +\|Profile\|Datasetslice\|Reasoning\|Resolved\| Score \| |
| 67 | +\|---\|---\|---\|---:\|---:\| |
| 68 | +\|Q6\_KGGUF\|\`SWE-bench/SWE-bench\_Verified\`test\`0:100\`\|on\|69/100\|69.0%\| |
| 69 | + |
| 70 | +Verified-miniquantsweep,50tasks: |
| 71 | + |
| 72 | +\|Quant/profile\|Resolvedchart\|Score\| |
| 73 | +\|---\|---\|---:\| |
| 74 | +\| Q6\_K \| \`##################################################################\` \| 66.0% \| |
| 75 | +\|Q3\_K\_M\|\`################################################################\`\|64.0% \| |
| 76 | +\|Q5\_K\_M\|\`################################################################\`\| 64.0%\| |
| 77 | +\|Q4\_K\_M\| \`##############################################################\` \| 62.0% \| |
| 78 | +\|IQ4\_XS\| \`############################################################\` \|60.0%\| |
| 79 | +\| IQ4\_XS-MTP-graft-headQ6\|\`############################################################\`\|60.0%\| |
| 80 | +\|Q8\_0 \| \`##########################################################\` \| 58.0% \| |
| 81 | + |
| 82 | +\## Scope |
| 83 | + |
| 84 | +\- ServingissingleGPUonly:\`tp=1\`,onemodelcopy per GPU. |
| 85 | +-GPU0is the servingspeed lane. |
| 86 | +-GPU1isanindependentexperimentlaneforquantization,fine-tuning,oraudit jobs. |
| 87 | +-Multi-GPUservingprofilesareintentionallyoutofscopefor this model version. |
| 88 | + |
| 89 | +##InitialTracks |
| 90 | + |
| 91 | +1. Baselineandtunesingle-GPUservingwithSGLang,vLLM,andllama.cppGGUF. |
| 92 | +2.Buildareusablemodelauditharnessfortokenizer,chattemplate,special-token,reasoning,andtool-callissues. |
| 93 | +3. EvaluateexistingGGUFquantizations,thentestsmallerorhigher-throughputformats. |
| 94 | +4. Prepareacoding-datafine-tuningandpost-trainingpipeline. |
| 95 | + |
| 96 | +Currentreusableworkflow:\[MODEL\_INVESTIGATION\_HARNESS.md\](MODEL\_INVESTIGATION\_HARNESS.md). |
| 97 | +Detailed quant/MTP/model-release report: \[MODEL\_RELEASE\_REPORT.md\](MODEL\_RELEASE\_REPORT.md). |
| 98 | +Pi-agentquantcodingtaskreport: |
| 99 | +\[benchmarks/pi-agent-flappy-quant-test.md\](benchmarks/pi-agent-flappy-quant-test.md) |
| 100 | +withrawevidence under \[probes/pi-agent-flappy/\](probes/pi-agent-flappy/). |
| 101 | +Dedicated MTP KLD/sequence evaluation: |
| 102 | +\[benchmarks/mtp-dedicated-kld-evaluation.md\](benchmarks/mtp-dedicated-kld-evaluation.md) |
| 103 | +withrawevidence under \[benchmarks/raw/\](benchmarks/raw/). |
| 104 | +SWE-bench agent evaluation: |
| 105 | +\[benchmarks/swebench-agent-evals.md\](benchmarks/swebench-agent-evals.md) |
| 106 | +withrawscorereportsunder\[benchmarks/raw/swebench/\](benchmarks/raw/swebench/). |
| 107 | +Currentrequirementaudit:\[PROJECT\_STATUS\_AUDIT.md\](PROJECT\_STATUS\_AUDIT.md). |
| 108 | +SFT remediation plan: \[SFT\_REMEDIATION\_PLAN.md\](SFT\_REMEDIATION\_PLAN.md). |
| 109 | +Servingprofilecatalog:\[configs/serving\_profiles.yaml\](configs/serving\_profiles.yaml) |
| 110 | +with validation report \[probes/serving-profile-validation.md\](probes/serving-profile-validation.md). |
| 111 | +Quantartifactcatalog:\[configs/quant\_artifacts.yaml\](configs/quant\_artifacts.yaml) |
| 112 | +withvalidationreport\[probes/quant-artifact-validation.md\](probes/quant-artifact-validation.md). |
| 113 | +MTPprofilecatalog:\[configs/mtp\_profiles.yaml\](configs/mtp\_profiles.yaml) |
| 114 | +withvalidationreport\[probes/mtp-profile-validation.md\](probes/mtp-profile-validation.md). |
| 115 | +Projectrequirementcatalog:\[configs/project\_requirements.yaml\](configs/project\_requirements.yaml) |
| 116 | +withvalidationreport\[probes/project-requirement-validation.md\](probes/project-requirement-validation.md). |
| 117 | +Idle-state policy: \[configs/idle\_state\_policy.yaml\](configs/idle\_state\_policy.yaml) |
| 118 | +withvalidationreport\[probes/idle-state-validation.md\](probes/idle-state-validation.md). |
| 119 | +CPU validation suite: \[scripts/run\_cpu\_validation\_suite.py\](scripts/run\_cpu\_validation\_suite.py) |
| 120 | +withreport \[probes/cpu-validation-suite.md\](probes/cpu-validation-suite.md). |
| 121 | +Goal completion audit: \[scripts/build\_goal\_completion\_audit.py\](scripts/build\_goal\_completion\_audit.py) |
| 122 | +withreport\[probes/goal-completion-audit.md\](probes/goal-completion-audit.md). |
| 123 | +Release-referenceconsistency:\[scripts/validate\_release\_references.py\](scripts/validate\_release\_references.py) |
| 124 | +withreport \[probes/release-reference-validation.md\](probes/release-reference-validation.md). |
| 125 | +Behaviorregressionmatrix:\[probes/behavior-regression-matrix.md\](probes/behavior-regression-matrix.md) |
| 126 | +via \[scripts/summarize\_behavior\_matrix.py\](scripts/summarize\_behavior\_matrix.py). |
| 127 | +Releasereadinessreport:\[probes/release-readiness-report.md\](probes/release-readiness-report.md) |
| 128 | +via \[scripts/build\_release\_readiness\_report.py\](scripts/build\_release\_readiness\_report.py). |
| 129 | +HFreleaseverification:\[probes/hf-release-verification.json\](probes/hf-release-verification.json) |
| 130 | +via\[scripts/verify\_hf\_release.py\](scripts/verify\_hf\_release.py).Latestlocal |
| 131 | +runpassed 48 checks with 0 failures and 3 expected incomplete-project warnings; |
| 132 | +the exact verified commit is recorded in \`remote.sha\` in the JSON report. |
| 133 | +Publishingaverifierreportcanleavethestagedpublishedreportonemetadata |
| 134 | +commit behind the newest remote head; \`scripts/validate\_release\_references.py\` |
| 135 | +and\`scripts/build\_goal\_completion\_audit.py\`recordbothpublishedandlatest |
| 136 | +localverifierevidencetopreventstalecompletionclaims. |
| 137 | +WhenthesamescriptrunsfromafreshHFrepoclone,\`hf\_upload/...\`defaults |
| 138 | +fallbacktorepo-rootpaths;amissinglocallatest-verifierrunisawarning, |
| 139 | +notafailure. |
| 140 | + |
| 141 | +##CurrentServingBaseline |
| 142 | + |
| 143 | +\- BestcurrentGGUFservingprofile:llama.cpp\`Q4\_K\_M\`\`tp=1\` with \`REASONING=off\`. |
| 144 | +\- Integrated MTP serving profile: \`IQ4\_XS-MTP-graft-headQ6\` with adaptive throttling, \`LLAMA\_SPEC\_MAX\_DRAFTING\_SLOTS=1\`; this keeps low-concurrency MTP speedups and avoids the saturated c16 loss. The MTP catalog validates all 8 published MTP GGUF artifacts. |
| 145 | +-Pi-agentFlappyBird quant test: all 7 tested profiles generated a runnable vanilla HTML/CSS/JS canvas game. \`IQ4\_XS-MTP-graft-headQ6\` was fastest on this agent workload at 320.91 decode tok/s, with 88.92%MTPdraft-tokenacceptance (\`3452/3882\`) andper-position acceptance \`(0.936, 0.843)\`. Q3\_K\_M passed only after 3 Pi-agentrepairpasses;allother profiles passed on the first run. |
| 146 | +-EarliervLLMBF16\`tp=1\` and SGLangBF16\`tp=1\`profilesarepreservedas baseline notes, but the publishable model package is the llama.cpp GGUF stack. |
| 147 | +\- The default llama.cppscriptuses\`REASONING=off\`becausethemodelotherwisespendssimplecodingpromptsin\`reasoning\_content\` before producing final \`content\`. |
| 148 | +-Custom Q3\_K\_M \`tp=1\` is now validated as the lowest-memoryGGUFoption: 15.61GiBon disk, 17.27 GiB loaded VRAM in the short-context benchmark profile, and 0 behavior-suite issues. Q4\_K\_M remains faster. |
| 149 | +\- Custom IQ4\_XS \`tp=1\` is validated as the middle-footprint GGUF option and is the base body for the integrated MTP graft artifact. |
| 150 | +\- vLLMGGUF Q4\_K\_M currently loads but has broken output and is not benchmark-valid. |
| 151 | +\- Profile notes: \`runs/vllm-initial-profile.md\` and \`runs/sglang-initial-profile.md\`. |
| 152 | +-Quantizedcontrol note: \`runs/llamacpp-q4-control.md\`. |
| 153 | +-GGUF/vLLMmaterializationaudit: \`runs/gguf-vllm-materialization-q4.md\`. |
| 154 | +\- Reusablebenchmarksummary: |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
| 155 |  |
| 156 | +\`\`\`bash |
| 157 | +.venv-lab/bin/pythonscripts/summarize\_benchmarks.pyruns/\*-seq-c\*-256.jsonl |
| 158 | +\`\`\` |
|  |  |
| 159 |  |
| 160 | +##GPU1ExperimentLane |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
| 161 |  |
| 162 | +\- Laneconfig: \`configs/experiment\_lanes.yaml\`. |
| 163 | +\- Quantization matrix: \`configs/quantization\_matrix.yaml\`. |
| 164 | +\- Custom Q3\_K\_M profile: \`runs/llamacpp-q3-k-m-profile.md\`. |
| 165 | +\- Training plan: \`configs/training\_plan.yaml\`. |
| 166 | +\- Source-backed run note: \`runs/gpu1-quant-train-plan.md\`. |
| 167 |  |
| 168 | +##HardwareBaseline |
|  |  |
|  |  |
|  |  |
|  |  |
| 169 |  |
| 170 | +-GPU0:NVIDIARTXPRO6000BlackwellMax-QWorkstationEdition,97,887MiBVRAM. |
| 171 | +- GPU1: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 97,887 MiB VRAM. |
| 172 | +\- Driver:580.159.03, CUDAruntimereportedbydriver:13.0. |
| 173 | +-Host had no \`nvcc\` andno\`cmake\`atprojectstart. Alocal \`.venv-build\`nowprovides CMake, Ninja, and CUDA 13.2wheelsfor the llama.cppCUDA build. |
|  |  |
| 174 |  |
| 175 | +##QuickStart |
| 176 |  |
| 177 | +Createthelightweightlabenvironment: |
|  |  |
|  |  |
| 178 |  |
| 179 | +\`\`\`bash |
| 180 | +cd /home/ripper/ornith-35b-lab |
| 181 | +uv venv .venv-lab --python 3.12 |
| 182 | +source .venv-lab/bin/activate |
| 183 | +uv pip install -r requirements-lab.txt |
| 184 | +\`\`\` |
| 185 |  |
| 186 | +Runthemetadataauditwithoutdownloadingfull model weights: |
| 187 |  |
| 188 | \`\`\`bash |
| 189 | +pythonscripts/audit\_model.py--outputruns/audit-ornith-35b.json |
| 190 | +.venv-lab/bin/python scripts/check\_audit\_gates.py runs/audit-ornith-35b.json |
| 191 | +\`\`\` |
| 192 | + |
| 193 | +Run the combined reusable investigation report: |
| 194 | + |
| 195 | +\`\`\`bash |
| 196 | +.venv-lab/bin/python scripts/run\_model\_investigation.py \ |
| 197 | + --output runs/investigation-ornith-metadata.json |
| 198 | +.venv-lab/bin/python scripts/summarize\_investigation.py \ |
| 199 | + runs/investigation-ornith-metadata.json \ |
| 200 | + --output runs/investigation-ornith-metadata.md |
| 201 | +\`\`\` |
| 202 | + |
| 203 | +Add \`--base-url\` and \`--served-model\` when a live OpenAI-compatible backend is |
| 204 | +running to include the behavior suite in the same report. |
| 205 | + |
| 206 | +Verify the published HF release and local staged support-file hashes: |
| 207 | + |
| 208 | +\`\`\`bash |
| 209 | +.venv-lab/bin/python scripts/verify\_hf\_release.py \ |
| 210 | + --output runs/hf-release-verification.json |
| 211 | +\`\`\` |
| 212 | + |
| 213 | +When running from a clone of the HF repo rather than this lab workspace, pass |
| 214 | +\`--local-dir . --manifest ARTIFACT\_MANIFEST.json\`. |
| 215 | + |
| 216 | +Run the local CPU-only validation suite: |
| 217 | + |
| 218 | +\`\`\`bash |
| 219 | +.venv-lab/bin/python scripts/run\_cpu\_validation\_suite.py \ |
| 220 | + --root . \ |
| 221 | + --output runs/cpu-validation-suite.json \ |
| 222 | + --markdown-output runs/cpu-validation-suite.md \ |
| 223 | + --fail-on-error |
| 224 | +\`\`\` |
| 225 | + |
| 226 | +This runs idle-state, serving profile, quant artifact, MTP profile, |
| 227 | +release-readiness, release-reference, project-requirement, and goal-completion validation in |
| 228 | +dependency order. It does not start model serving, SFT, quantization, or eval |
| 229 | +workloads. Its summary includes both failed steps and report-level warnings so |
| 230 | +expected incomplete-project warnings stay visible without failing the suite. |
| 231 | + |
| 232 | +Build the CPU-only release-readiness report: |
| 233 | + |
| 234 | +\`\`\`bash |
| 235 | +.venv-lab/bin/python scripts/build\_release\_readiness\_report.py \ |
| 236 | + --allow-unreleased-sft \ |
| 237 | + --serving-profile-validation hf\_upload/probes/serving-profile-validation.json \ |
| 238 | + --quant-artifact-validation hf\_upload/probes/quant-artifact-validation.json \ |
| 239 | + --mtp-profile-validation hf\_upload/probes/mtp-profile-validation.json \ |
| 240 | + --required-doc hf\_upload/README.md \ |
| 241 | + --required-doc hf\_upload/RELEASE\_AUDIT.md \ |
| 242 | + --required-doc hf\_upload/PROJECT\_STATUS\_AUDIT.md \ |
| 243 | + --required-doc hf\_upload/MODEL\_INVESTIGATION\_HARNESS.md \ |
| 244 | + --required-doc hf\_upload/SFT\_REMEDIATION\_PLAN.md \ |
| 245 | + --output runs/release-readiness-report.json \ |
| 246 | + --markdown-output runs/release-readiness-report.md \ |
| 247 | + --fail-on-serving-not-ready |
| 248 | +\`\`\` |
| 249 | + |
| 250 | +The current readiness report passes the GGUF serving release and marks the full |
| 251 | +project incomplete because no SFT/post-training adapter is released. |
| 252 | + |
| 253 | +Validate the machine-readable serving profiles: |
| 254 | + |
| 255 | +\`\`\`bash |
| 256 | +.venv-lab/bin/python scripts/validate\_serving\_profiles.py \ |
| 257 | + --catalog configs/serving\_profiles.yaml \ |
| 258 | + --manifest hf\_upload/ARTIFACT\_MANIFEST.json \ |
| 259 | + --hf-root hf\_upload \ |
| 260 | + --output runs/serving-profile-validation.json \ |
| 261 | + --markdown-output runs/serving-profile-validation.md \ |
| 262 | + --fail-on-error |
| 263 | +\`\`\` |
| 264 | + |
| 265 | +Validate the machine-readable quant artifacts: |
| 266 | + |
| 267 | +\`\`\`bash |
| 268 | +.venv-lab/bin/python scripts/validate\_quant\_artifacts.py \ |
| 269 | + --catalog configs/quant\_artifacts.yaml \ |
| 270 | + --manifest hf\_upload/ARTIFACT\_MANIFEST.json \ |
| 271 | + --hf-root hf\_upload \ |
| 272 | + --output runs/quant-artifact-validation.json \ |
| 273 | + --markdown-output runs/quant-artifact-validation.md \ |
| 274 | + --fail-on-error |
| 275 | +\`\`\` |
| 276 | + |
| 277 | +Validate the machine-readable MTP profiles: |
| 278 | + |
| 279 | +\`\`\`bash |
| 280 | +.venv-lab/bin/python scripts/validate\_mtp\_profiles.py \ |
| 281 | + --catalog configs/mtp\_profiles.yaml \ |
| 282 | + --manifest hf\_upload/ARTIFACT\_MANIFEST.json \ |
| 283 | + --hf-root hf\_upload \ |
| 284 | + --output runs/mtp-profile-validation.json \ |
| 285 | + --markdown-output runs/mtp-profile-validation.md \ |
| 286 | + --fail-on-error |
| 287 | +\`\`\` |
| 288 | + |
| 289 | +Validate the project-level requirement status: |
| 290 | + |
| 291 | +\`\`\`bash |
| 292 | +.venv-lab/bin/python scripts/validate\_project\_requirements.py \ |
| 293 | + --catalog configs/project\_requirements.yaml \ |
| 294 | + --root . \ |
| 295 | + --output runs/project-requirement-validation.json \ |
| 296 | + --markdown-output runs/project-requirement-validation.md \ |
| 297 | + --fail-on-error |
| 298 | +\`\`\` |
| 299 | + |
| 300 | +This report intentionally separates \`serving\_release\_status=pass\` from |
| 301 | +\`full\_project\_status=incomplete\` until a release-candidate SFT/post-training |
| 302 | +adapter exists. |
| 303 | + |
| 304 | +Validate the stopped/idle runtime state: |
| 305 | + |
| 306 | +\`\`\`bash |
| 307 | +.venv-lab/bin/python scripts/validate\_idle\_state.py \ |
| 308 | + --config configs/idle\_state\_policy.yaml \ |
| 309 | + --root . \ |
| 310 | + --output runs/idle-state-validation.json \ |
| 311 | + --markdown-output runs/idle-state-validation.md \ |
| 312 | + --fail-on-error |
| 313 | +\`\`\` |
| 314 | + |
| 315 | +This is a live local guard: it checks the saved SFT stop record, \`nvidia-smi\` |
| 316 | +GPU utilization and compute apps, and known training/server/quant process names. |
| 317 | + |
| 318 | +Run reusable behavior probes against any OpenAI-compatible backend: |
| 319 | + |
| 320 | +\`\`\`bash |
| 321 | +.venv-lab/bin/python scripts/probe\_model\_behavior.py \ |
| 322 | + --base-url http://localhost:8000/v1 \ |
| 323 | + --model Ornith-1.0-35B \ |
| 324 | + --output runs/behavior-vllm-bf16.json |
| 325 | +\`\`\` |
| 326 | + |
| 327 | +The suite lives in \`configs/model\_behavior\_suite.yaml\` and is designed to be |
| 328 | +copied to other models. It turns model bugs into named, repeatable checks for |
| 329 | +coding correctness, JSON following, chat-template leakage, decode corruption, |
| 330 | +and optional tool-call handling. |
| 331 | + |
| 332 | +Run the GGUF/vLLM materialization audit for a quantized artifact: |
| 333 | + |
| 334 | +\`\`\`bash |
| 335 | +.venv-lab/bin/python scripts/audit\_gguf\_vllm\_materialization.py \ |
| 336 | + --model deepreinforce-ai/Ornith-1.0-35B \ |
| 337 | + --gguf deepreinforce-ai/Ornith-1.0-35B-GGUF:Q4\_K\_M \ |
| 338 | + --output runs/gguf-vllm-materialization-q4.json |
| 339 | \`\`\` |
| 340 |  |
| 341 | +StartoneservingbackendonGPU0afterinstallingthatbackend in its own environment: |
|  |  |
| 342 |  |
| 343 | \`\`\`bash |
| 344 | +CUDA\_VISIBLE\_DEVICES=0scripts/serve\_vllm\_gpu0.sh |
| 345 | +CUDA\_VISIBLE\_DEVICES=0 scripts/serve\_sglang\_gpu0.sh |
| 346 | +CUDA\_VISIBLE\_DEVICES=0 scripts/serve\_llamacpp\_gpu0.sh |
| 347 | +\`\`\` |
| 348 | + |
| 349 | +Serve the custom local Q3\_K\_M GGUF: |
| 350 | + |
| 351 | +\`\`\`bash |
| 352 | +QUANT=Q3\_K\_M PORT=8002 CTX\_SIZE=8192 PARALLEL=16 \ |
| 353 | scripts/serve\_llamacpp\_gpu0.sh |
| 354 | \`\`\` |
| 355 |  |
| 356 | +RunasmokebenchmarkagainstanOpenAI-compatibleserver: |
| 357 | + |
| 358 | +\`\`\`bash |
| 359 | +pythonscripts/bench\_openai.py\ |
| 360 | + --base-url http://localhost:8000/v1 \ |
| 361 | + --model Ornith-1.0-35B \ |
| 362 | + --concurrency 1 \ |
| 363 | + --max-tokens 256 \ |
| 364 | + --outputruns/smoke-bench.jsonl |
| 365 | +\`\`\` |
| 366 | + |
| 367 | +ProbetheOpenAI-compatibleservercontractaftereachbackend/profilechange: |
| 368 | + |
| 369 | +\`\`\`bash |
| 370 | +.venv-lab/bin/pythonscripts/probe\_openai\_contract.py\ |
| 371 | + --base-urlhttp://localhost:8000/v1 \ |
| 372 | + --model Ornith-1.0-35B \ |
| 373 | +--outputruns/server-contract.json |
| 374 | +\`\`\` |
| 375 | + |
| 376 | +Prepare and gate SFT data before launching GPU training: |
| 377 | + |
| 378 | +\`\`\`bash |
| 379 | +.venv-train/bin/python scripts/render\_sft\_jsonl.py \ |
| 380 | +--inputdata/train/coding\_sft\_messages.jsonl \ |
| 381 | +--outputdata/train/coding\_sft.jsonl |
| 382 | + |
| 383 | +.venv-train/bin/pythonscripts/validate\_sft\_jsonl.py \ |
| 384 | +--train-jsonldata/train/coding\_sft.jsonl \ |
| 385 | +--eval-jsonldata/eval/coding\_sft\_eval.jsonl\ |
| 386 | + --output runs/sft-data-validation.json |
| 387 | +\`\`\` |
| 388 | + |
| 389 | +\`scripts/train\_sft\_lora.py\`runsthesamevalidationbeforeimportingTorchand |
| 390 | +loadingthe35Bmodelunless\`--skip-data-validation\`ispassed. |
| 391 | + |
| 392 | +TheproductionSFTrun was stopped at user request. Latest durable checkpoint: |
| 393 | +\`artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint-8000\`; stop record: |
| 394 | +\`runs/ornith-35b-coding-lora-prod-20k.stopped.json\`. |
| 395 | + |
| 396 | +CPU-side SFT remediation is prepared but not running. The repair path is defined |
| 397 | +in \`configs/sft\_repair\_blend.yaml\` and uses: |
| 398 | + |
| 399 | +\- \`scripts/build\_sft\_repair\_set.py\` |
| 400 | +\- \`scripts/build\_sft\_code\_only\_from\_fences.py\` |
| 401 | +\- \`scripts/build\_sft\_blend.py\` |
| 402 | +\- \`data/train/coding\_sft\_prod\_repair\_strong.jsonl\` |
| 403 | +\- \`data/eval/coding\_sft\_prod\_repair\_strong\_eval.jsonl\` |
| 404 | +\- \`runs/sft-data-validation-prod-repair-strong.json\` |
| 405 | +\- \`runs/sft-format-audit-prod-repair-strong.json\` |
| 406 | +\- \`runs/sft-format-gate-code-only-extract.json\` |
| 407 | +\- \`runs/sft-format-gate-prod-repair-strong.json\` |
| 408 | +\- \`runs/sft-resume-preflight-strong.json\` |
| 409 | +\- \`runs/sft-format-gate-strict-repair.json\` |
| 410 | +\- \`runs/sft-format-gate-prod-repair.json\` |
| 411 | + |
| 412 | +The recommended strong repair split has 44,096 train rows, 2,312 eval rows, and |
| 413 | +no blocking validation issues. It reduces train message code fences to 54.43% |
| 414 | +and \`Here's\`/\`Here is\` preambles to 30.49%. The code-only extraction gate passes |
| 415 | +8/8 checks, and the strong blended repair gate passes 10/10 checks. Do not |
| 416 | +resume SFT unless explicitly requested. |
| 417 | + |
| 418 | +Before any future resume, run the report-only guard: |
| 419 | + |
| 420 | +\`\`\`bash |
| 421 | +.venv-lab/bin/python scripts/preflight\_sft\_resume.py \ |
| 422 | + --output runs/sft-resume-preflight-strong.json |
| 423 | +\`\`\` |
| 424 | + |
| 425 | +The current preflight passes 16/16 checks and includes the exact resume command |
| 426 | +preview without launching training. |
| 427 | + |
| 428 | +Run the production SFT monitor only for a resumed run: |
| 429 | + |
| 430 | +\`\`\`bash |
| 431 | +scripts/monitor\_sft\_prod.sh |
| 432 | +\`\`\` |
| 433 | + |
| 434 | +Start the automatic checkpoint behavior gate watcher only for a resumed run: |
| 435 | + |
| 436 | +\`\`\`bash |
| 437 | +scripts/watch\_sft\_checkpoint\_gate.sh |
| 438 | +\`\`\` |
| 439 | + |
| 440 | +Evaluate a LoRA checkpoint against the reusable behavior suite: |
| 441 | + |
| 442 | +\`\`\`bash |
| 443 | +CUDA\_VISIBLE\_DEVICES=0 \ |
| 444 | +.venv-train/bin/python scripts/eval\_lora\_behavior.py \ |
| 445 | + --base-model deepreinforce-ai/Ornith-1.0-35B \ |
| 446 | + --adapter artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint-<step> \ |
| 447 | + --suite configs/model\_behavior\_suite.yaml \ |
| 448 | + --output runs/eval-lora-behavior-checkpoint-<step>.json |
| 449 | +\`\`\` |
| 450 | + |
| 451 | +The LoRA evaluator defaults to final-answer mode, matching llama.cpp |
| 452 | +\`REASONING=off\`. Add \`--enable-thinking\` only for explicit reasoning-mode tests. |

benchmarks/raw/swebench/ornith-q6-think-verified100-summary.jsonADDED
Viewed

|     |     |
| --- | --- |
|  | @@ -0,0 +1,67 @@ |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

|     |     |
| --- | --- |
|  |  |
| 1 | +{ |
| 2 | + "run": "ornith\_q6\_think\_verified100", |
| 3 | + "date": "2026-06-29", |
| 4 | + "model": "Ornith-1.0-35B Q6\_K GGUF", |
| 5 | + "reasoning": { |
| 6 | + "enabled": true, |
| 7 | + "enable\_thinking": true, |
| 8 | + "preserve\_thinking": true, |
| 9 | + "auto\_disable\_thinking\_with\_tools": false, |
| 10 | + "explicit\_thinking\_budget": null |
| 11 | + }, |
| 12 | + "sampling": { |
| 13 | + "temperature": 1.0, |
| 14 | + "top\_p": 1.0, |
| 15 | + "max\_output\_tokens": 8192 |
| 16 | + }, |
| 17 | + "dataset": { |
| 18 | + "name": "SWE-bench/SWE-bench\_Verified", |
| 19 | + "split": "test", |
| 20 | + "slice": "0:100" |
| 21 | + }, |
| 22 | + "score": { |
| 23 | + "resolved": 69, |
| 24 | + "total": 100, |
| 25 | + "resolved\_pct": 69.0, |
| 26 | + "unresolved": 25, |
| 27 | + "empty\_patch\_or\_timeout": 6, |
| 28 | + "infra\_errors\_after\_rerun": 0 |
| 29 | + }, |
| 30 | + "original\_harness\_report": { |
| 31 | + "path": "/home/ripper/dsv4-local/eval/swebench/reports/ornith\_q6\_think\_verified100.json", |
| 32 | + "resolved": 69, |
| 33 | + "unresolved": 23, |
| 34 | + "empty\_patch": 6, |
| 35 | + "infra\_errors": 2, |
| 36 | + "note": "The wrapper's total\_instances=500 / metric\_value=13.80 is not the intended 100-task slice score because the first harness invocation did not pass instance filters." |
| 37 | + }, |
| 38 | + "rerun\_report\_for\_original\_errors": { |
| 39 | + "path": "/home/ripper/dsv4-local/eval/swebench/reports/ornith\_q6\_think\_verified100\_rerun\_errors.json", |
| 40 | + "instances": \[ |\
| 41 | + "astropy\_\_astropy-8707", |\
| 42 | + "astropy\_\_astropy-8872" |\
| 43 | + \], |
| 44 | + "resolved": 0, |
| 45 | + "unresolved": 2, |
| 46 | + "infra\_errors": 0 |
| 47 | + }, |
| 48 | + "prediction\_artifacts": { |
| 49 | + "merged\_predictions": "/home/ripper/ornith-35b-lab/runs/reasoning-verified100-20260629/predictions/q6\_think\_verified100\_merged/preds.json", |
| 50 | + "merge\_manifest": "/home/ripper/ornith-35b-lab/runs/reasoning-verified100-20260629/predictions/q6\_think\_verified100\_merged/merge\_manifest.json" |
| 51 | + }, |
| 52 | + "trace\_reliability": { |
| 53 | + "path": "/home/ripper/ornith-35b-lab/runs/reasoning-verified100-20260629/trace\_reliability\_q6\_think\_verified100.json", |
| 54 | + "trajectory\_count": 100, |
| 55 | + "assistant\_turns": 6821, |
| 56 | + "assistant\_tool\_call\_turns": 6821, |
| 57 | + "assistant\_no\_tool\_call\_turns": 0, |
| 58 | + "finish\_reason\_length\_turns": 0, |
| 59 | + "format\_error\_no\_tool\_calls": 23, |
| 60 | + "duplicate\_tool\_call\_turns": 2, |
| 61 | + "hidden\_reasoning\_repeat\_turns": 0, |
| 62 | + "visible\_think\_tag\_turns": 1, |
| 63 | + "visible\_tool\_tag\_turns": 1, |
| 64 | + "submitted": 94, |
| 65 | + "time\_exceeded": 6 |
| 66 | + } |
| 67 | +} |

benchmarks/raw/swebench/ornith\_iq4\_xs\_mtp\_graft\_verified\_mini.jsonADDED
Viewed

|     |     |
| --- | --- |
|  | @@ -0,0 +1,170 @@ |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

|     |     |
| --- | --- |
|  |  |
| 1 | +{ |
| 2 | + "total\_instances": 50, |
| 3 | + "submitted\_instances": 50, |
| 4 | + "completed\_instances": 48, |
| 5 | + "resolved\_instances": 30, |
| 6 | + "unresolved\_instances": 18, |
| 7 | + "empty\_patch\_instances": 2, |
| 8 | + "error\_instances": 0, |
| 9 | + "completed\_ids": \[ |\
| 10 | + "django\_\_django-11790", |\
| 11 | + "django\_\_django-11815", |\
| 12 | + "django\_\_django-11848", |\
| 13 | + "django\_\_django-11880", |\
| 14 | + "django\_\_django-11885", |\
| 15 | + "django\_\_django-11951", |\
| 16 | + "django\_\_django-11964", |\
| 17 | + "django\_\_django-11999", |\
| 18 | + "django\_\_django-12039", |\
| 19 | + "django\_\_django-12050", |\
| 20 | + "django\_\_django-12143", |\
| 21 | + "django\_\_django-12155", |\
| 22 | + "django\_\_django-12193", |\
| 23 | + "django\_\_django-12209", |\
| 24 | + "django\_\_django-12262", |\
| 25 | + "django\_\_django-12273", |\
| 26 | + "django\_\_django-12276", |\
| 27 | + "django\_\_django-12304", |\
| 28 | + "django\_\_django-12308", |\
| 29 | + "django\_\_django-12325", |\
| 30 | + "django\_\_django-12406", |\
| 31 | + "django\_\_django-12708", |\
| 32 | + "django\_\_django-12713", |\
| 33 | + "django\_\_django-12774", |\
| 34 | + "django\_\_django-9296", |\
| 35 | + "sphinx-doc\_\_sphinx-10323", |\
| 36 | + "sphinx-doc\_\_sphinx-10435", |\
| 37 | + "sphinx-doc\_\_sphinx-10466", |\
| 38 | + "sphinx-doc\_\_sphinx-10673", |\
| 39 | + "sphinx-doc\_\_sphinx-11510", |\
| 40 | + "sphinx-doc\_\_sphinx-7590", |\
| 41 | + "sphinx-doc\_\_sphinx-7748", |\
| 42 | + "sphinx-doc\_\_sphinx-7757", |\
| 43 | + "sphinx-doc\_\_sphinx-7985", |\
| 44 | + "sphinx-doc\_\_sphinx-8035", |\
| 45 | + "sphinx-doc\_\_sphinx-8056", |\
| 46 | + "sphinx-doc\_\_sphinx-8265", |\
| 47 | + "sphinx-doc\_\_sphinx-8269", |\
| 48 | + "sphinx-doc\_\_sphinx-8475", |\
| 49 | + "sphinx-doc\_\_sphinx-8548", |\
| 50 | + "sphinx-doc\_\_sphinx-8551", |\
| 51 | + "sphinx-doc\_\_sphinx-8638", |\
| 52 | + "sphinx-doc\_\_sphinx-8721", |\
| 53 | + "sphinx-doc\_\_sphinx-9230", |\
| 54 | + "sphinx-doc\_\_sphinx-9281", |\
| 55 | + "sphinx-doc\_\_sphinx-9320", |\
| 56 | + "sphinx-doc\_\_sphinx-9367", |\
| 57 | + "sphinx-doc\_\_sphinx-9698" |\
| 58 | + \], |
| 59 | + "incomplete\_ids": \[\], |
| 60 | + "empty\_patch\_ids": \[ |\
| 61 | + "sphinx-doc\_\_sphinx-9229", |\
| 62 | + "sphinx-doc\_\_sphinx-9461" |\
| 63 | + \], |
| 64 | + "submitted\_ids": \[ |\
| 65 | + "django\_\_django-11790", |\
| 66 | + "django\_\_django-11815", |\
| 67 | + "django\_\_django-11848", |\
| 68 | + "django\_\_django-11880", |\
| 69 | + "django\_\_django-11885", |\
| 70 | + "django\_\_django-11951", |\
| 71 | + "django\_\_django-11964", |\
| 72 | + "django\_\_django-11999", |\
| 73 | + "django\_\_django-12039", |\
| 74 | + "django\_\_django-12050", |\
| 75 | + "django\_\_django-12143", |\
| 76 | + "django\_\_django-12155", |\
| 77 | + "django\_\_django-12193", |\
| 78 | + "django\_\_django-12209", |\
| 79 | + "django\_\_django-12262", |\
| 80 | + "django\_\_django-12273", |\
| 81 | + "django\_\_django-12276", |\
| 82 | + "django\_\_django-12304", |\
| 83 | + "django\_\_django-12308", |\
| 84 | + "django\_\_django-12325", |\
| 85 | + "django\_\_django-12406", |\
| 86 | + "django\_\_django-12708", |\
| 87 | + "django\_\_django-12713", |\
| 88 | + "django\_\_django-12774", |\
| 89 | + "django\_\_django-9296", |\
| 90 | + "sphinx-doc\_\_sphinx-10323", |\
| 91 | + "sphinx-doc\_\_sphinx-10435", |\
| 92 | + "sphinx-doc\_\_sphinx-10466", |\
| 93 | + "sphinx-doc\_\_sphinx-10673", |\
| 94 | + "sphinx-doc\_\_sphinx-11510", |\
| 95 | + "sphinx-doc\_\_sphinx-7590", |\
| 96 | + "sphinx-doc\_\_sphinx-7748", |\
| 97 | + "sphinx-doc\_\_sphinx-7757", |\
| 98 | + "sphinx-doc\_\_sphinx-7985", |\
| 99 | + "sphinx-doc\_\_sphinx-8035", |\
| 100 | + "sphinx-doc\_\_sphinx-8056", |\
| 101 | + "sphinx-doc\_\_sphinx-8265", |\
| 102 | + "sphinx-doc\_\_sphinx-8269", |\
| 103 | + "sphinx-doc\_\_sphinx-8475", |\
| 104 | + "sphinx-doc\_\_sphinx-8548", |\
| 105 | + "sphinx-doc\_\_sphinx-8551", |\
| 106 | + "sphinx-doc\_\_sphinx-8638", |\
| 107 | + "sphinx-doc\_\_sphinx-8721", |\
| 108 | + "sphinx-doc\_\_sphinx-9229", |\
| 109 | + "sphinx-doc\_\_sphinx-9230", |\
| 110 | + "sphinx-doc\_\_sphinx-9281", |\
| 111 | + "sphinx-doc\_\_sphinx-9320", |\
| 112 | + "sphinx-doc\_\_sphinx-9367", |\
| 113 | + "sphinx-doc\_\_sphinx-9461", |\
| 114 | + "sphinx-doc\_\_sphinx-9698" |\
| 115 | + \], |
| 116 | + "resolved\_ids": \[ |\
| 117 | + "django\_\_django-11815", |\
| 118 | + "django\_\_django-11880", |\
| 119 | + "django\_\_django-11951", |\
| 120 | + "django\_\_django-11964", |\
| 121 | + "django\_\_django-11999", |\
| 122 | + "django\_\_django-12039", |\
| 123 | + "django\_\_django-12050", |\
| 124 | + "django\_\_django-12143", |\
| 125 | + "django\_\_django-12155", |\
| 126 | + "django\_\_django-12193", |\
| 127 | + "django\_\_django-12209", |\
| 128 | + "django\_\_django-12262", |\
| 129 | + "django\_\_django-12276", |\
| 130 | + "django\_\_django-12304", |\
| 131 | + "django\_\_django-12708", |\
| 132 | + "django\_\_django-12713", |\
| 133 | + "django\_\_django-12774", |\
| 134 | + "django\_\_django-9296", |\
| 135 | + "sphinx-doc\_\_sphinx-10466", |\
| 136 | + "sphinx-doc\_\_sphinx-7757", |\
| 137 | + "sphinx-doc\_\_sphinx-8035", |\
| 138 | + "sphinx-doc\_\_sphinx-8269", |\
| 139 | + "sphinx-doc\_\_sphinx-8475", |\
| 140 | + "sphinx-doc\_\_sphinx-8551", |\
| 141 | + "sphinx-doc\_\_sphinx-8721", |\
| 142 | + "sphinx-doc\_\_sphinx-9230", |\
| 143 | + "sphinx-doc\_\_sphinx-9281", |\
| 144 | + "sphinx-doc\_\_sphinx-9320", |\
| 145 | + "sphinx-doc\_\_sphinx-9367", |\
| 146 | + "sphinx-doc\_\_sphinx-9698" |\
| 147 | + \], |
| 148 | + "unresolved\_ids": \[ |\
| 149 | + "django\_\_django-11790", |\
| 150 | + "django\_\_django-11848", |\
| 151 | + "django\_\_django-11885", |\
| 152 | + "django\_\_django-12273", |\
| 153 | + "django\_\_django-12308", |\
| 154 | + "django\_\_django-12325", |\
| 155 | + "django\_\_django-12406", |\
| 156 | + "sphinx-doc\_\_sphinx-10323", |\
| 157 | + "sphinx-doc\_\_sphinx-10435", |\
| 158 | + "sphinx-doc\_\_sphinx-10673", |\
| 159 | + "sphinx-doc\_\_sphinx-11510", |\
| 160 | + "sphinx-doc\_\_sphinx-7590", |\
| 161 | + "sphinx-doc\_\_sphinx-7748", |\
| 162 | + "sphinx-doc\_\_sphinx-7985", |\
| 163 | + "sphinx-doc\_\_sphinx-8056", |\
| 164 | + "sphinx-doc\_\_sphinx-8265", |\
| 165 | + "sphinx-doc\_\_sphinx-8548", |\
| 166 | + "sphinx-doc\_\_sphinx-8638" |\
| 167 | + \], |
| 168 | + "error\_ids": \[\], |
| 169 | + "schema\_version": 2 |
| 170 | +} |

benchmarks/raw/swebench/ornith\_iq4\_xs\_verified\_mini.jsonADDED
Viewed

|     |     |
| --- | --- |
|  | @@ -0,0 +1,167 @@ |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

|     |     |
| --- | --- |
|  |  |
| 1 | +{ |
| 2 | + "total\_instances": 50, |
| 3 | + "submitted\_instances": 50, |
| 4 | + "completed\_instances": 45, |
| 5 | + "resolved\_instances": 30, |
| 6 | + "unresolved\_instances": 15, |
| 7 | + "empty\_patch\_instances": 5, |
| 8 | + "error\_instances": 0, |
| 9 | + "completed\_ids": \[ |\
| 10 | + "django\_\_django-11790", |\
| 11 | + "django\_\_django-11815", |\
| 12 | + "django\_\_django-11848", |\
| 13 | + "django\_\_django-11880", |\
| 14 | + "django\_\_django-11885", |\
| 15 | + "django\_\_django-11951", |\
| 16 | + "django\_\_django-11999", |\
| 17 | + "django\_\_django-12039", |\
| 18 | + "django\_\_django-12050", |\
| 19 | + "django\_\_django-12143", |\
| 20 | + "django\_\_django-12155", |\
| 21 | + "django\_\_django-12193", |\
| 22 | + "django\_\_django-12262", |\
| 23 | + "django\_\_django-12273", |\
| 24 | + "django\_\_django-12276", |\
| 25 | + "django\_\_django-12304", |\
| 26 | + "django\_\_django-12308", |\
| 27 | + "django\_\_django-12325", |\
| 28 | + "django\_\_django-12406", |\
| 29 | + "django\_\_django-12708", |\
| 30 | + "django\_\_django-12713", |\
| 31 | + "django\_\_django-12774", |\
| 32 | + "django\_\_django-9296", |\
| 33 | + "sphinx-doc\_\_sphinx-10323", |\
| 34 | + "sphinx-doc\_\_sphinx-10435", |\
| 35 | + "sphinx-doc\_\_sphinx-10466", |\
| 36 | + "sphinx-doc\_\_sphinx-11510", |\
| 37 | + "sphinx-doc\_\_sphinx-7590", |\
| 38 | + "sphinx-doc\_\_sphinx-7748", |\
| 39 | + "sphinx-doc\_\_sphinx-7757", |\
| 40 | + "sphinx-doc\_\_sphinx-7985", |\
| 41 | + "sphinx-doc\_\_sphinx-8035", |\
| 42 | + "sphinx-doc\_\_sphinx-8056", |\
| 43 | + "sphinx-doc\_\_sphinx-8265", |\
| 44 | + "sphinx-doc\_\_sphinx-8269", |\
| 45 | + "sphinx-doc\_\_sphinx-8475", |\
| 46 | + "sphinx-doc\_\_sphinx-8548", |\
| 47 | + "sphinx-doc\_\_sphinx-8551", |\
| 48 | + "sphinx-doc\_\_sphinx-8638", |\
| 49 | + "sphinx-doc\_\_sphinx-8721", |\
| 50 | + "sphinx-doc\_\_sphinx-9230", |\
| 51 | + "sphinx-doc\_\_sphinx-9281", |\
| 52 | + "sphinx-doc\_\_sphinx-9320", |\
| 53 | + "sphinx-doc\_\_sphinx-9367", |\
| 54 | + "sphinx-doc\_\_sphinx-9698" |\
| 55 | + \], |
| 56 | + "incomplete\_ids": \[\], |
| 57 | + "empty\_patch\_ids": \[ |\
| 58 | + "django\_\_django-11964", |\
| 59 | + "django\_\_django-12209", |\
| 60 | + "sphinx-doc\_\_sphinx-10673", |\
| 61 | + "sphinx-doc\_\_sphinx-9229", |\
| 62 | + "sphinx-doc\_\_sphinx-9461" |\
| 63 | + \], |
| 64 | + "submitted\_ids": \[ |\
| 65 | + "django\_\_django-11790", |\
| 66 | + "django\_\_django-11815", |\
| 67 | + "django\_\_django-11848", |\
| 68 | + "django\_\_django-11880", |\
| 69 | + "django\_\_django-11885", |\
| 70 | + "django\_\_django-11951", |\
| 71 | + "django\_\_django-11964", |\
| 72 | + "django\_\_django-11999", |\
| 73 | + "django\_\_django-12039", |\
| 74 | + "django\_\_django-12050", |\
| 75 | + "django\_\_django-12143", |\
| 76 | + "django\_\_django-12155", |\
| 77 | + "django\_\_django-12193", |\
| 78 | + "django\_\_django-12209", |\
| 79 | + "django\_\_django-12262", |\
| 80 | + "django\_\_django-12273", |\
| 81 | + "django\_\_django-12276", |\
| 82 | + "django\_\_django-12304", |\
| 83 | + "django\_\_django-12308", |\
| 84 | + "django\_\_django-12325", |\
| 85 | + "django\_\_django-12406", |\
| 86 | + "django\_\_django-12708", |\
| 87 | + "django\_\_django-12713", |\
| 88 | + "django\_\_django-12774", |\
| 89 | + "django\_\_django-9296", |\
| 90 | + "sphinx-doc\_\_sphinx-10323", |\
| 91 | + "sphinx-doc\_\_sphinx-10435", |\
| 92 | + "sphinx-doc\_\_sphinx-10466", |\
| 93 | + "sphinx-doc\_\_sphinx-10673", |\
| 94 | + "sphinx-doc\_\_sphinx-11510", |\
| 95 | + "sphinx-doc\_\_sphinx-7590", |\
| 96 | + "sphinx-doc\_\_sphinx-7748", |\
| 97 | + "sphinx-doc\_\_sphinx-7757", |\
| 98 | + "sphinx-doc\_\_sphinx-7985", |\
| 99 | + "sphinx-doc\_\_sphinx-8035", |\
| 100 | + "sphinx-doc\_\_sphinx-8056", |\
| 101 | + "sphinx-doc\_\_sphinx-8265", |\
| 102 | + "sphinx-doc\_\_sphinx-8269", |\
| 103 | + "sphinx-doc\_\_sphinx-8475", |\
| 104 | + "sphinx-doc\_\_sphinx-8548", |\
| 105 | + "sphinx-doc\_\_sphinx-8551", |\
| 106 | + "sphinx-doc\_\_sphinx-8638", |\
| 107 | + "sphinx-doc\_\_sphinx-8721", |\
| 108 | + "sphinx-doc\_\_sphinx-9229", |\
| 109 | + "sphinx-doc\_\_sphinx-9230", |\
| 110 | + "sphinx-doc\_\_sphinx-9281", |\
| 111 | + "sphinx-doc\_\_sphinx-9320", |\
| 112 | + "sphinx-doc\_\_sphinx-9367", |\
| 113 | + "sphinx-doc\_\_sphinx-9461", |\
| 114 | + "sphinx-doc\_\_sphinx-9698" |\
| 115 | + \], |
| 116 | + "resolved\_ids": \[ |\
| 117 | + "django\_\_django-11790", |\
| 118 | + "django\_\_django-11848", |\
| 119 | + "django\_\_django-11880", |\
| 120 | + "django\_\_django-11951", |\
| 121 | + "django\_\_django-11999", |\
| 122 | + "django\_\_django-12039", |\
| 123 | + "django\_\_django-12050", |\
| 124 | + "django\_\_django-12143", |\
| 125 | + "django\_\_django-12155", |\
| 126 | + "django\_\_django-12262", |\
| 127 | + "django\_\_django-12276", |\
| 128 | + "django\_\_django-12304", |\
| 129 | + "django\_\_django-12708", |\
| 130 | + "django\_\_django-12713", |\
| 131 | + "django\_\_django-12774", |\
| 132 | + "django\_\_django-9296", |\
| 133 | + "sphinx-doc\_\_sphinx-10466", |\
| 134 | + "sphinx-doc\_\_sphinx-7757", |\
| 135 | + "sphinx-doc\_\_sphinx-8035", |\
| 136 | + "sphinx-doc\_\_sphinx-8265", |\
| 137 | + "sphinx-doc\_\_sphinx-8269", |\
| 138 | + "sphinx-doc\_\_sphinx-8475", |\
| 139 | + "sphinx-doc\_\_sphinx-8548", |\
| 140 | + "sphinx-doc\_\_sphinx-8551", |\
| 141 | + "sphinx-doc\_\_sphinx-8721", |\
| 142 | + "sphinx-doc\_\_sphinx-9230", |\
| 143 | + "sphinx-doc\_\_sphinx-9281", |\
| 144 | + "sphinx-doc\_\_sphinx-9320", |\
| 145 | + "sphinx-doc\_\_sphinx-9367", |\
| 146 | + "sphinx-doc\_\_sphinx-9698" |\
| 147 | + \], |
| 148 | + "unresolved\_ids": \[ |\
| 149 | + "django\_\_django-11815", |\
| 150 | + "django\_\_django-11885", |\
| 151 | + "django\_\_django-12193", |\
| 152 | + "django\_\_django-12273", |\
| 153 | + "django\_\_django-12308", |\
| 154 | + "django\_\_django-12325", |\
| 155 | + "django\_\_django-12406", |\
| 156 | + "sphinx-doc\_\_sphinx-10323", |\
| 157 | + "sphinx-doc\_\_sphinx-10435", |\
| 158 | + "sphinx-doc\_\_sphinx-11510", |\
| 159 | + "sphinx-doc\_\_sphinx-7590", |\
| 160 | + "sphinx-doc\_\_sphinx-7748", |\
| 161 | + "sphinx-doc\_\_sphinx-7985", |\
| 162 | + "sphinx-doc\_\_sphinx-8056", |\
| 163 | + "sphinx-doc\_\_sphinx-8638" |\
| 164 | + \], |
| 165 | + "error\_ids": \[\], |
| 166 | + "schema\_version": 2 |
| 167 | +} |

benchmarks/raw/swebench/ornith\_q3\_k\_m\_verified\_mini.jsonADDED
Viewed

|     |     |
| --- | --- |
|  | @@ -0,0 +1,169 @@ |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

|     |     |
| --- | --- |
|  |  |
| 1 | +{ |
| 2 | + "total\_instances": 50, |
| 3 | + "submitted\_instances": 50, |
| 4 | + "completed\_instances": 47, |
| 5 | + "resolved\_instances": 32, |
| 6 | + "unresolved\_instances": 15, |
| 7 | + "empty\_patch\_instances": 3, |
| 8 | + "error\_instances": 0, |
| 9 | + "completed\_ids": \[ |\
| 10 | + "django\_\_django-11790", |\
| 11 | + "django\_\_django-11815", |\
| 12 | + "django\_\_django-11848", |\
| 13 | + "django\_\_django-11880", |\
| 14 | + "django\_\_django-11951", |\
| 15 | + "django\_\_django-11964", |\
| 16 | + "django\_\_django-11999", |\
| 17 | + "django\_\_django-12039", |\
| 18 | + "django\_\_django-12050", |\
| 19 | + "django\_\_django-12143", |\
| 20 | + "django\_\_django-12155", |\
| 21 | + "django\_\_django-12193", |\
| 22 | + "django\_\_django-12209", |\
| 23 | + "django\_\_django-12262", |\
| 24 | + "django\_\_django-12276", |\
| 25 | + "django\_\_django-12304", |\
| 26 | + "django\_\_django-12308", |\
| 27 | + "django\_\_django-12325", |\
| 28 | + "django\_\_django-12406", |\
| 29 | + "django\_\_django-12708", |\
| 30 | + "django\_\_django-12713", |\
| 31 | + "django\_\_django-12774", |\
| 32 | + "django\_\_django-9296", |\
| 33 | + "sphinx-doc\_\_sphinx-10323", |\
| 34 | + "sphinx-doc\_\_sphinx-10435", |\
| 35 | + "sphinx-doc\_\_sphinx-10466", |\
| 36 | + "sphinx-doc\_\_sphinx-10673", |\
| 37 | + "sphinx-doc\_\_sphinx-11510", |\
| 38 | + "sphinx-doc\_\_sphinx-7590", |\
| 39 | + "sphinx-doc\_\_sphinx-7748", |\
| 40 | + "sphinx-doc\_\_sphinx-7757", |\
| 41 | + "sphinx-doc\_\_sphinx-7985", |\
| 42 | + "sphinx-doc\_\_sphinx-8035", |\
| 43 | + "sphinx-doc\_\_sphinx-8056", |\
| 44 | + "sphinx-doc\_\_sphinx-8265", |\
| 45 | + "sphinx-doc\_\_sphinx-8269", |\
| 46 | + "sphinx-doc\_\_sphinx-8475", |\
| 47 | + "sphinx-doc\_\_sphinx-8548", |\
| 48 | + "sphinx-doc\_\_sphinx-8551", |\
| 49 | + "sphinx-doc\_\_sphinx-8638", |\
| 50 | + "sphinx-doc\_\_sphinx-8721", |\
| 51 | + "sphinx-doc\_\_sphinx-9230", |\
| 52 | + "sphinx-doc\_\_sphinx-9281", |\
| 53 | + "sphinx-doc\_\_sphinx-9320", |\
| 54 | + "sphinx-doc\_\_sphinx-9367", |\
| 55 | + "sphinx-doc\_\_sphinx-9461", |\
| 56 | + "sphinx-doc\_\_sphinx-9698" |\
| 57 | + \], |
| 58 | + "incomplete\_ids": \[\], |
| 59 | + "empty\_patch\_ids": \[ |\
| 60 | + "django\_\_django-11885", |\
| 61 | + "django\_\_django-12273", |\
| 62 | + "sphinx-doc\_\_sphinx-9229" |\
| 63 | + \], |
| 64 | + "submitted\_ids": \[ |\
| 65 | + "django\_\_django-11790", |\
| 66 | + "django\_\_django-11815", |\
| 67 | + "django\_\_django-11848", |\
| 68 | + "django\_\_django-11880", |\
| 69 | + "django\_\_django-11885", |\
| 70 | + "django\_\_django-11951", |\
| 71 | + "django\_\_django-11964", |\
| 72 | + "django\_\_django-11999", |\
| 73 | + "django\_\_django-12039", |\
| 74 | + "django\_\_django-12050", |\
| 75 | + "django\_\_django-12143", |\
| 76 | + "django\_\_django-12155", |\
| 77 | + "django\_\_django-12193", |\
| 78 | + "django\_\_django-12209", |\
| 79 | + "django\_\_django-12262", |\
| 80 | + "django\_\_django-12273", |\
| 81 | + "django\_\_django-12276", |\
| 82 | + "django\_\_django-12304", |\
| 83 | + "django\_\_django-12308", |\
| 84 | + "django\_\_django-12325", |\
| 85 | + "django\_\_django-12406", |\
| 86 | + "django\_\_django-12708", |\
| 87 | + "django\_\_django-12713", |\
| 88 | + "django\_\_django-12774", |\
| 89 | + "django\_\_django-9296", |\
| 90 | + "sphinx-doc\_\_sphinx-10323", |\
| 91 | + "sphinx-doc\_\_sphinx-10435", |\
| 92 | + "sphinx-doc\_\_sphinx-10466", |\
| 93 | + "sphinx-doc\_\_sphinx-10673", |\
| 94 | + "sphinx-doc\_\_sphinx-11510", |\
| 95 | + "sphinx-doc\_\_sphinx-7590", |\
| 96 | + "sphinx-doc\_\_sphinx-7748", |\
| 97 | + "sphinx-doc\_\_sphinx-7757", |\
| 98 | + "sphinx-doc\_\_sphinx-7985", |\
| 99 | + "sphinx-doc\_\_sphinx-8035", |\
| 100 | + "sphinx-doc\_\_sphinx-8056", |\
| 101 | + "sphinx-doc\_\_sphinx-8265", |\
| 102 | + "sphinx-doc\_\_sphinx-8269", |\
| 103 | + "sphinx-doc\_\_sphinx-8475", |\
| 104 | + "sphinx-doc\_\_sphinx-8548", |\
| 105 | + "sphinx-doc\_\_sphinx-8551", |\
| 106 | + "sphinx-doc\_\_sphinx-8638", |\
| 107 | + "sphinx-doc\_\_sphinx-8721", |\
| 108 | + "sphinx-doc\_\_sphinx-9229", |\
| 109 | + "sphinx-doc\_\_sphinx-9230", |\
| 110 | + "sphinx-doc\_\_sphinx-9281", |\
| 111 | + "sphinx-doc\_\_sphinx-9320", |\
| 112 | + "sphinx-doc\_\_sphinx-9367", |\
| 113 | + "sphinx-doc\_\_sphinx-9461", |\
| 114 | + "sphinx-doc\_\_sphinx-9698" |\
| 115 | + \], |
| 116 | + "resolved\_ids": \[ |\
| 117 | + "django\_\_django-11815", |\
| 118 | + "django\_\_django-11848", |\
| 119 | + "django\_\_django-11880", |\
| 120 | + "django\_\_django-11951", |\
| 121 | + "django\_\_django-11964", |\
| 122 | + "django\_\_django-11999", |\
| 123 | + "django\_\_django-12039", |\
| 124 | + "django\_\_django-12050", |\
| 125 | + "django\_\_django-12143", |\
| 126 | + "django\_\_django-12155", |\
| 127 | + "django\_\_django-12193", |\
| 128 | + "django\_\_django-12209", |\
| 129 | + "django\_\_django-12262", |\
| 130 | + "django\_\_django-12276", |\
| 131 | + "django\_\_django-12304", |\
| 132 | + "django\_\_django-12708", |\
| 133 | + "django\_\_django-12713", |\
| 134 | + "django\_\_django-12774", |\
| 135 | + "django\_\_django-9296", |\
| 136 | + "sphinx-doc\_\_sphinx-10466", |\
| 137 | + "sphinx-doc\_\_sphinx-10673", |\
| 138 | + "sphinx-doc\_\_sphinx-7757", |\
| 139 | + "sphinx-doc\_\_sphinx-8035", |\
| 140 | + "sphinx-doc\_\_sphinx-8265", |\
| 141 | + "sphinx-doc\_\_sphinx-8269", |\
| 142 | + "sphinx-doc\_\_sphinx-8475", |\
| 143 | + "sphinx-doc\_\_sphinx-8551", |\
| 144 | + "sphinx-doc\_\_sphinx-8721", |\
| 145 | + "sphinx-doc\_\_sphinx-9281", |\
| 146 | + "sphinx-doc\_\_sphinx-9320", |\
| 147 | + "sphinx-doc\_\_sphinx-9367", |\
| 148 | + "sphinx-doc\_\_sphinx-9698" |\
| 149 | + \], |
| 150 | + "unresolved\_ids": \[ |\
| 151 | + "django\_\_django-11790", |\
| 152 | + "django\_\_django-12308", |\
| 153 | + "django\_\_django-12325", |\
| 154 | + "django\_\_django-12406", |\
| 155 | + "sphinx-doc\_\_sphinx-10323", |\
| 156 | + "sphinx-doc\_\_sphinx-10435", |\
| 157 | + "sphinx-doc\_\_sphinx-11510", |\
| 158 | + "sphinx-doc\_\_sphinx-7590", |\
| 159 | + "sphinx-doc\_\_sphinx-7748", |\
| 160 | + "sphinx-doc\_\_sphinx-7985", |\
| 161 | + "sphinx-doc\_\_sphinx-8056", |\
| 162 | + "sphinx-doc\_\_sphinx-8548", |\
| 163 | + "sphinx-doc\_\_sphinx-8638", |\
| 164 | + "sphinx-doc\_\_sphinx-9230", |\
| 165 | + "sphinx-doc\_\_sphinx-9461" |\
| 166 | + \], |
| 167 | + "error\_ids": \[\], |
| 168 | + "schema\_version": 2 |
| 169 | +} |

benchmarks/raw/swebench/ornith\_q4\_k\_m\_verified\_mini.jsonADDED
Viewed

|     |     |
| --- | --- |
|  | @@ -0,0 +1,170 @@ |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

|     |     |
| --- | --- |
|  |  |
| 1 | +{ |
| 2 | + "total\_instances": 50, |
| 3 | + "submitted\_instances": 50, |
| 4 | + "completed\_instances": 48, |
| 5 | + "resolved\_instances": 31, |
| 6 | + "unresolved\_instances": 17, |
| 7 | + "empty\_patch\_instances": 2, |
| 8 | + "error\_instances": 0, |
| 9 | + "completed\_ids": \[ |\
| 10 | + "django\_\_django-11790", |\
| 11 | + "django\_\_django-11815", |\
| 12 | + "django\_\_django-11848", |\
| 13 | + "django\_\_django-11880", |\
| 14 | + "django\_\_django-11885", |\
| 15 | + "django\_\_django-11951", |\
| 16 | + "django\_\_django-11964", |\
| 17 | + "django\_\_django-11999", |\
| 18 | + "django\_\_django-12039", |\
| 19 | + "django\_\_django-12050", |\
| 20 | + "django\_\_django-12143", |\
| 21 | + "django\_\_django-12155", |\
| 22 | + "django\_\_django-12193", |\
| 23 | + "django\_\_django-12262", |\
| 24 | + "django\_\_django-12273", |\
| 25 | + "django\_\_django-12276", |\
| 26 | + "django\_\_django-12304", |\
| 27 | + "django\_\_django-12308", |\
| 28 | + "django\_\_django-12325", |\
| 29 | + "django\_\_django-12406", |\
| 30 | + "django\_\_django-12708", |\
| 31 | + "django\_\_django-12713", |\
| 32 | + "django\_\_django-12774", |\
| 33 | + "django\_\_django-9296", |\
| 34 | + "sphinx-doc\_\_sphinx-10323", |\
| 35 | + "sphinx-doc\_\_sphinx-10435", |\
| 36 | + "sphinx-doc\_\_sphinx-10466", |\
| 37 | + "sphinx-doc\_\_sphinx-10673", |\
| 38 | + "sphinx-doc\_\_sphinx-11510", |\
| 39 | + "sphinx-doc\_\_sphinx-7590", |\
| 40 | + "sphinx-doc\_\_sphinx-7748", |\
| 41 | + "sphinx-doc\_\_sphinx-7757", |\
| 42 | + "sphinx-doc\_\_sphinx-7985", |\
| 43 | + "sphinx-doc\_\_sphinx-8035", |\
| 44 | + "sphinx-doc\_\_sphinx-8056", |\
| 45 | + "sphinx-doc\_\_sphinx-8265", |\
| 46 | + "sphinx-doc\_\_sphinx-8269", |\
| 47 | + "sphinx-doc\_\_sphinx-8475", |\
| 48 | + "sphinx-doc\_\_sphinx-8548", |\
| 49 | + "sphinx-doc\_\_sphinx-8551", |\
| 50 | + "sphinx-doc\_\_sphinx-8638", |\
| 51 | + "sphinx-doc\_\_sphinx-8721", |\
| 52 | + "sphinx-doc\_\_sphinx-9230", |\
| 53 | + "sphinx-doc\_\_sphinx-9281", |\
| 54 | + "sphinx-doc\_\_sphinx-9320", |\
| 55 | + "sphinx-doc\_\_sphinx-9367", |\
| 56 | + "sphinx-doc\_\_sphinx-9461", |\
| 57 | + "sphinx-doc\_\_sphinx-9698" |\
| 58 | + \], |
| 59 | + "incomplete\_ids": \[\], |
| 60 | + "empty\_patch\_ids": \[ |\
| 61 | + "django\_\_django-12209", |\
| 62 | + "sphinx-doc\_\_sphinx-9229" |\
| 63 | + \], |
| 64 | + "submitted\_ids": \[ |\
| 65 | + "django\_\_django-11790", |\
| 66 | + "django\_\_django-11815", |\
| 67 | + "django\_\_django-11848", |\
| 68 | + "django\_\_django-11880", |\
| 69 | + "django\_\_django-11885", |\
| 70 | + "django\_\_django-11951", |\
| 71 | + "django\_\_django-11964", |\
| 72 | + "django\_\_django-11999", |\
| 73 | + "django\_\_django-12039", |\
| 74 | + "django\_\_django-12050", |\
| 75 | + "django\_\_django-12143", |\
| 76 | + "django\_\_django-12155", |\
| 77 | + "django\_\_django-12193", |\
| 78 | + "django\_\_django-12209", |\
| 79 | + "django\_\_django-12262", |\
| 80 | + "django\_\_django-12273", |\
| 81 | + "django\_\_django-12276", |\
| 82 | + "django\_\_django-12304", |\
| 83 | + "django\_\_django-12308", |\
| 84 | + "django\_\_django-12325", |\
| 85 | + "django\_\_django-12406", |\
| 86 | + "django\_\_django-12708", |\
| 87 | + "django\_\_django-12713", |\
| 88 | + "django\_\_django-12774", |\
| 89 | + "django\_\_django-9296", |\
| 90 | + "sphinx-doc\_\_sphinx-10323", |\
| 91 | + "sphinx-doc\_\_sphinx-10435", |\
| 92 | + "sphinx-doc\_\_sphinx-10466", |\
| 93 | + "sphinx-doc\_\_sphinx-10673", |\
| 94 | + "sphinx-doc\_\_sphinx-11510", |\
| 95 | + "sphinx-doc\_\_sphinx-7590", |\
| 96 | + "sphinx-doc\_\_sphinx-7748", |\
| 97 | + "sphinx-doc\_\_sphinx-7757", |\
| 98 | + "sphinx-doc\_\_sphinx-7985", |\
| 99 | + "sphinx-doc\_\_sphinx-8035", |\
| 100 | + "sphinx-doc\_\_sphinx-8056", |\
| 101 | + "sphinx-doc\_\_sphinx-8265", |\
| 102 | + "sphinx-doc\_\_sphinx-8269", |\
| 103 | + "sphinx-doc\_\_sphinx-8475", |\
| 104 | + "sphinx-doc\_\_sphinx-8548", |\
| 105 | + "sphinx-doc\_\_sphinx-8551", |\
| 106 | + "sphinx-doc\_\_sphinx-8638", |\
| 107 | + "sphinx-doc\_\_sphinx-8721", |\
| 108 | + "sphinx-doc\_\_sphinx-9229", |\
| 109 | + "sphinx-doc\_\_sphinx-9230", |\
| 110 | + "sphinx-doc\_\_sphinx-9281", |\
| 111 | + "sphinx-doc\_\_sphinx-9320", |\
| 112 | + "sphinx-doc\_\_sphinx-9367", |\
| 113 | + "sphinx-doc\_\_sphinx-9461", |\
| 114 | + "sphinx-doc\_\_sphinx-9698" |\
| 115 | + \], |
| 116 | + "resolved\_ids": \[ |\
| 117 | + "django\_\_django-11815", |\
| 118 | + "django\_\_django-11848", |\
| 119 | + "django\_\_django-11880", |\
| 120 | + "django\_\_django-11951", |\
| 121 | + "django\_\_django-11964", |\
| 122 | + "django\_\_django-11999", |\
| 123 | + "django\_\_django-12039", |\
| 124 | + "django\_\_django-12050", |\
| 125 | + "django\_\_django-12143", |\
| 126 | + "django\_\_django-12155", |\
| 127 | + "django\_\_django-12262", |\
| 128 | + "django\_\_django-12273", |\
| 129 | + "django\_\_django-12276", |\
| 130 | + "django\_\_django-12304", |\
| 131 | + "django\_\_django-12708", |\
| 132 | + "django\_\_django-12713", |\
| 133 | + "django\_\_django-12774", |\
| 134 | + "django\_\_django-9296", |\
| 135 | + "sphinx-doc\_\_sphinx-10466", |\
| 136 | + "sphinx-doc\_\_sphinx-7757", |\
| 137 | + "sphinx-doc\_\_sphinx-8035", |\
| 138 | + "sphinx-doc\_\_sphinx-8269", |\
| 139 | + "sphinx-doc\_\_sphinx-8475", |\
| 140 | + "sphinx-doc\_\_sphinx-8548", |\
| 141 | + "sphinx-doc\_\_sphinx-8551", |\
| 142 | + "sphinx-doc\_\_sphinx-8721", |\
| 143 | + "sphinx-doc\_\_sphinx-9230", |\
| 144 | + "sphinx-doc\_\_sphinx-9281", |\
| 145 | + "sphinx-doc\_\_sphinx-9320", |\
| 146 | + "sphinx-doc\_\_sphinx-9367", |\
| 147 | + "sphinx-doc\_\_sphinx-9698" |\
| 148 | + \], |
| 149 | + "unresolved\_ids": \[ |\
| 150 | + "django\_\_django-11790", |\
| 151 | + "django\_\_django-11885", |\
| 152 | + "django\_\_django-12193", |\
| 153 | + "django\_\_django-12308", |\
| 154 | + "django\_\_django-12325", |\
| 155 | + "django\_\_django-12406", |\
| 156 | + "sphinx-doc\_\_sphinx-10323", |\
| 157 | + "sphinx-doc\_\_sphinx-10435", |\
| 158 | + "sphinx-doc\_\_sphinx-10673", |\
| 159 | + "sphinx-doc\_\_sphinx-11510", |\
| 160 | + "sphinx-doc\_\_sphinx-7590", |\
| 161 | + "sphinx-doc\_\_sphinx-7748", |\
| 162 | + "sphinx-doc\_\_sphinx-7985", |\
| 163 | + "sphinx-doc\_\_sphinx-8056", |\
| 164 | + "sphinx-doc\_\_sphinx-8265", |\
| 165 | + "sphinx-doc\_\_sphinx-8638", |\
| 166 | + "sphinx-doc\_\_sphinx-9461" |\
| 167 | + \], |
| 168 | + "error\_ids": \[\], |
| 169 | + "schema\_version": 2 |
| 170 | +} |

benchmarks/raw/swebench/ornith\_q5\_k\_m\_verified\_mini.jsonADDED
Viewed

|     |     |
| --- | --- |
|  | @@ -0,0 +1,171 @@ |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

|     |     |
| --- | --- |
|  |  |
| 1 | +{ |
| 2 | + "total\_instances": 50, |
| 3 | + "submitted\_instances": 50, |
| 4 | + "completed\_instances": 49, |
| 5 | + "resolved\_instances": 32, |
| 6 | + "unresolved\_instances": 17, |
| 7 | + "empty\_patch\_instances": 1, |
| 8 | + "error\_instances": 0, |
| 9 | + "completed\_ids": \[ |\
| 10 | + "django\_\_django-11790", |\
| 11 | + "django\_\_django-11815", |\
| 12 | + "django\_\_django-11848", |\
| 13 | + "django\_\_django-11880", |\
| 14 | + "django\_\_django-11885", |\
| 15 | + "django\_\_django-11951", |\
| 16 | + "django\_\_django-11964", |\
| 17 | + "django\_\_django-11999", |\
| 18 | + "django\_\_django-12039", |\
| 19 | + "django\_\_django-12050", |\
| 20 | + "django\_\_django-12143", |\
| 21 | + "django\_\_django-12155", |\
| 22 | + "django\_\_django-12193", |\
| 23 | + "django\_\_django-12209", |\
| 24 | + "django\_\_django-12262", |\
| 25 | + "django\_\_django-12273", |\
| 26 | + "django\_\_django-12276", |\
| 27 | + "django\_\_django-12304", |\
| 28 | + "django\_\_django-12308", |\
| 29 | + "django\_\_django-12325", |\
| 30 | + "django\_\_django-12406", |\
| 31 | + "django\_\_django-12708", |\
| 32 | + "django\_\_django-12713", |\
| 33 | + "django\_\_django-12774", |\
| 34 | + "django\_\_django-9296", |\
| 35 | + "sphinx-doc\_\_sphinx-10323", |\
| 36 | + "sphinx-doc\_\_sphinx-10435", |\
| 37 | + "sphinx-doc\_\_sphinx-10466", |\
| 38 | + "sphinx-doc\_\_sphinx-10673", |\
| 39 | + "sphinx-doc\_\_sphinx-11510", |\
| 40 | + "sphinx-doc\_\_sphinx-7590", |\
| 41 | + "sphinx-doc\_\_sphinx-7748", |\
| 42 | + "sphinx-doc\_\_sphinx-7757", |\
| 43 | + "sphinx-doc\_\_sphinx-7985", |\
| 44 | + "sphinx-doc\_\_sphinx-8035", |\
| 45 | + "sphinx-doc\_\_sphinx-8056", |\
| 46 | + "sphinx-doc\_\_sphinx-8265", |\
| 47 | + "sphinx-doc\_\_sphinx-8269", |\
| 48 | + "sphinx-doc\_\_sphinx-8475", |\
| 49 | + "sphinx-doc\_\_sphinx-8548", |\
| 50 | + "sphinx-doc\_\_sphinx-8551", |\
| 51 | + "sphinx-doc\_\_sphinx-8638", |\
| 52 | + "sphinx-doc\_\_sphinx-8721", |\
| 53 | + "sphinx-doc\_\_sphinx-9230", |\
| 54 | + "sphinx-doc\_\_sphinx-9281", |\
| 55 | + "sphinx-doc\_\_sphinx-9320", |\
| 56 | + "sphinx-doc\_\_sphinx-9367", |\
| 57 | + "sphinx-doc\_\_sphinx-9461", |\
| 58 | + "sphinx-doc\_\_sphinx-9698" |\
| 59 | + \], |
| 60 | + "incomplete\_ids": \[\], |
| 61 | + "empty\_patch\_ids": \[ |\
| 62 | + "sphinx-doc\_\_sphinx-9229" |\
| 63 | + \], |
| 64 | + "submitted\_ids": \[ |\
| 65 | + "django\_\_django-11790", |\
| 66 | + "django\_\_django-11815", |\
| 67 | + "django\_\_django-11848", |\
| 68 | + "django\_\_django-11880", |\
| 69 | + "django\_\_django-11885", |\
| 70 | + "django\_\_django-11951", |\
| 71 | + "django\_\_django-11964", |\
| 72 | + "django\_\_django-11999", |\
| 73 | + "django\_\_django-12039", |\
| 74 | + "django\_\_django-12050", |\
| 75 | + "django\_\_django-12143", |\
| 76 | + "django\_\_django-12155", |\
| 77 | + "django\_\_django-12193", |\
| 78 | + "django\_\_django-12209", |\
| 79 | + "django\_\_django-12262", |\
| 80 | + "django\_\_django-12273", |\
| 81 | + "django\_\_django-12276", |\
| 82 | + "django\_\_django-12304", |\
| 83 | + "django\_\_django-12308", |\
| 84 | + "django\_\_django-12325", |\
| 85 | + "django\_\_django-12406", |\
| 86 | + "django\_\_django-12708", |\
| 87 | + "django\_\_django-12713", |\
| 88 | + "django\_\_django-12774", |\
| 89 | + "django\_\_django-9296", |\
| 90 | + "sphinx-doc\_\_sphinx-10323", |\
| 91 | + "sphinx-doc\_\_sphinx-10435", |\
| 92 | + "sphinx-doc\_\_sphinx-10466", |\
| 93 | + "sphinx-doc\_\_sphinx-10673", |\
| 94 | + "sphinx-doc\_\_sphinx-11510", |\
| 95 | + "sphinx-doc\_\_sphinx-7590", |\
| 96 | + "sphinx-doc\_\_sphinx-7748", |\
| 97 | + "sphinx-doc\_\_sphinx-7757", |\
| 98 | + "sphinx-doc\_\_sphinx-7985", |\
| 99 | + "sphinx-doc\_\_sphinx-8035", |\
| 100 | + "sphinx-doc\_\_sphinx-8056", |\
| 101 | + "sphinx-doc\_\_sphinx-8265", |\
| 102 | + "sphinx-doc\_\_sphinx-8269", |\
| 103 | + "sphinx-doc\_\_sphinx-8475", |\
| 104 | + "sphinx-doc\_\_sphinx-8548", |\
| 105 | + "sphinx-doc\_\_sphinx-8551", |\
| 106 | + "sphinx-doc\_\_sphinx-8638", |\
| 107 | + "sphinx-doc\_\_sphinx-8721", |\
| 108 | + "sphinx-doc\_\_sphinx-9229", |\
| 109 | + "sphinx-doc\_\_sphinx-9230", |\
| 110 | + "sphinx-doc\_\_sphinx-9281", |\
| 111 | + "sphinx-doc\_\_sphinx-9320", |\
| 112 | + "sphinx-doc\_\_sphinx-9367", |\
| 113 | + "sphinx-doc\_\_sphinx-9461", |\
| 114 | + "sphinx-doc\_\_sphinx-9698" |\
| 115 | + \], |
| 116 | + "resolved\_ids": \[ |\
| 117 | + "django\_\_django-11815", |\
| 118 | + "django\_\_django-11848", |\
| 119 | + "django\_\_django-11880", |\
| 120 | + "django\_\_django-11951", |\
| 121 | + "django\_\_django-11964", |\
| 122 | + "django\_\_django-11999", |\
| 123 | + "django\_\_django-12039", |\
| 124 | + "django\_\_django-12050", |\
| 125 | + "django\_\_django-12143", |\
| 126 | + "django\_\_django-12155", |\
| 127 | + "django\_\_django-12209", |\
| 128 | + "django\_\_django-12262", |\
| 129 | + "django\_\_django-12276", |\
| 130 | + "django\_\_django-12304", |\
| 131 | + "django\_\_django-12708", |\
| 132 | + "django\_\_django-12713", |\
| 133 | + "django\_\_django-12774", |\
| 134 | + "django\_\_django-9296", |\
| 135 | + "sphinx-doc\_\_sphinx-10466", |\
| 136 | + "sphinx-doc\_\_sphinx-10673", |\
| 137 | + "sphinx-doc\_\_sphinx-7757", |\
| 138 | + "sphinx-doc\_\_sphinx-8035", |\
| 139 | + "sphinx-doc\_\_sphinx-8265", |\
| 140 | + "sphinx-doc\_\_sphinx-8269", |\
| 141 | + "sphinx-doc\_\_sphinx-8475", |\
| 142 | + "sphinx-doc\_\_sphinx-8551", |\
| 143 | + "sphinx-doc\_\_sphinx-8721", |\
| 144 | + "sphinx-doc\_\_sphinx-9230", |\
| 145 | + "sphinx-doc\_\_sphinx-9281", |\
| 146 | + "sphinx-doc\_\_sphinx-9320", |\
| 147 | + "sphinx-doc\_\_sphinx-9367", |\
| 148 | + "sphinx-doc\_\_sphinx-9698" |\
| 149 | + \], |
| 150 | + "unresolved\_ids": \[ |\
| 151 | + "django\_\_django-11790", |\
| 152 | + "django\_\_django-11885", |\
| 153 | + "django\_\_django-12193", |\
| 154 | + "django\_\_django-12273", |\
| 155 | + "django\_\_django-12308", |\
| 156 | + "django\_\_django-12325", |\
| 157 | + "django\_\_django-12406", |\
| 158 | + "sphinx-doc\_\_sphinx-10323", |\
| 159 | + "sphinx-doc\_\_sphinx-10435", |\
| 160 | + "sphinx-doc\_\_sphinx-11510", |\
| 161 | + "sphinx-doc\_\_sphinx-7590", |\
| 162 | + "sphinx-doc\_\_sphinx-7748", |\
| 163 | + "sphinx-doc\_\_sphinx-7985", |\
| 164 | + "sphinx-doc\_\_sphinx-8056", |\
| 165 | + "sphinx-doc\_\_sphinx-8548", |\
| 166 | + "sphinx-doc\_\_sphinx-8638", |\
| 167 | + "sphinx-doc\_\_sphinx-9461" |\
| 168 | + \], |
| 169 | + "error\_ids": \[\], |
| 170 | + "schema\_version": 2 |
| 171 | +} |

benchmarks/raw/swebench/ornith\_q6\_k\_verified\_mini.jsonADDED
Viewed

|     |     |
| --- | --- |
|  | @@ -0,0 +1,170 @@ |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

|     |     |
| --- | --- |
|  |  |
| 1 | +{ |
| 2 | + "total\_instances": 50, |
| 3 | + "submitted\_instances": 50, |
| 4 | + "completed\_instances": 48, |
| 5 | + "resolved\_instances": 33, |
| 6 | + "unresolved\_instances": 15, |
| 7 | + "empty\_patch\_instances": 2, |
| 8 | + "error\_instances": 0, |
| 9 | + "completed\_ids": \[ |\
| 10 | + "django\_\_django-11790", |\
| 11 | + "django\_\_django-11815", |\
| 12 | + "django\_\_django-11848", |\
| 13 | + "django\_\_django-11880", |\
| 14 | + "django\_\_django-11951", |\
| 15 | + "django\_\_django-11964", |\
| 16 | + "django\_\_django-11999", |\
| 17 | + "django\_\_django-12039", |\
| 18 | + "django\_\_django-12050", |\
| 19 | + "django\_\_django-12143", |\
| 20 | + "django\_\_django-12155", |\
| 21 | + "django\_\_django-12193", |\
| 22 | + "django\_\_django-12209", |\
| 23 | + "django\_\_django-12262", |\
| 24 | + "django\_\_django-12273", |\
| 25 | + "django\_\_django-12276", |\
| 26 | + "django\_\_django-12304", |\
| 27 | + "django\_\_django-12308", |\
| 28 | + "django\_\_django-12325", |\
| 29 | + "django\_\_django-12406", |\
| 30 | + "django\_\_django-12708", |\
| 31 | + "django\_\_django-12713", |\
| 32 | + "django\_\_django-12774", |\
| 33 | + "django\_\_django-9296", |\
| 34 | + "sphinx-doc\_\_sphinx-10323", |\
| 35 | + "sphinx-doc\_\_sphinx-10435", |\
| 36 | + "sphinx-doc\_\_sphinx-10466", |\
| 37 | + "sphinx-doc\_\_sphinx-10673", |\
| 38 | + "sphinx-doc\_\_sphinx-11510", |\
| 39 | + "sphinx-doc\_\_sphinx-7590", |\
| 40 | + "sphinx-doc\_\_sphinx-7748", |\
| 41 | + "sphinx-doc\_\_sphinx-7757", |\
| 42 | + "sphinx-doc\_\_sphinx-7985", |\
| 43 | + "sphinx-doc\_\_sphinx-8035", |\
| 44 | + "sphinx-doc\_\_sphinx-8056", |\
| 45 | + "sphinx-doc\_\_sphinx-8265", |\
| 46 | + "sphinx-doc\_\_sphinx-8269", |\
| 47 | + "sphinx-doc\_\_sphinx-8475", |\
| 48 | + "sphinx-doc\_\_sphinx-8548", |\
| 49 | + "sphinx-doc\_\_sphinx-8551", |\
| 50 | + "sphinx-doc\_\_sphinx-8638", |\
| 51 | + "sphinx-doc\_\_sphinx-8721", |\
| 52 | + "sphinx-doc\_\_sphinx-9230", |\
| 53 | + "sphinx-doc\_\_sphinx-9281", |\
| 54 | + "sphinx-doc\_\_sphinx-9320", |\
| 55 | + "sphinx-doc\_\_sphinx-9367", |\
| 56 | + "sphinx-doc\_\_sphinx-9461", |\
| 57 | + "sphinx-doc\_\_sphinx-9698" |\
| 58 | + \], |
| 59 | + "incomplete\_ids": \[\], |
| 60 | + "empty\_patch\_ids": \[ |\
| 61 | + "django\_\_django-11885", |\
| 62 | + "sphinx-doc\_\_sphinx-9229" |\
| 63 | + \], |
| 64 | + "submitted\_ids": \[ |\
| 65 | + "django\_\_django-11790", |\
| 66 | + "django\_\_django-11815", |\
| 67 | + "django\_\_django-11848", |\
| 68 | + "django\_\_django-11880", |\
| 69 | + "django\_\_django-11885", |\
| 70 | + "django\_\_django-11951", |\
| 71 | + "django\_\_django-11964", |\
| 72 | + "django\_\_django-11999", |\
| 73 | + "django\_\_django-12039", |\
| 74 | + "django\_\_django-12050", |\
| 75 | + "django\_\_django-12143", |\
| 76 | + "django\_\_django-12155", |\
| 77 | + "django\_\_django-12193", |\
| 78 | + "django\_\_django-12209", |\
| 79 | + "django\_\_django-12262", |\
| 80 | + "django\_\_django-12273", |\
| 81 | + "django\_\_django-12276", |\
| 82 | + "django\_\_django-12304", |\
| 83 | + "django\_\_django-12308", |\
| 84 | + "django\_\_django-12325", |\
| 85 | + "django\_\_django-12406", |\
| 86 | + "django\_\_django-12708", |\
| 87 | + "django\_\_django-12713", |\
| 88 | + "django\_\_django-12774", |\
| 89 | + "django\_\_django-9296", |\
| 90 | + "sphinx-doc\_\_sphinx-10323", |\
| 91 | + "sphinx-doc\_\_sphinx-10435", |\
| 92 | + "sphinx-doc\_\_sphinx-10466", |\
| 93 | + "sphinx-doc\_\_sphinx-10673", |\
| 94 | + "sphinx-doc\_\_sphinx-11510", |\
| 95 | + "sphinx-doc\_\_sphinx-7590", |\
| 96 | + "sphinx-doc\_\_sphinx-7748", |\
| 97 | + "sphinx-doc\_\_sphinx-7757", |\
| 98 | + "sphinx-doc\_\_sphinx-7985", |\
| 99 | + "sphinx-doc\_\_sphinx-8035", |\
| 100 | + "sphinx-doc\_\_sphinx-8056", |\
| 101 | + "sphinx-doc\_\_sphinx-8265", |\
| 102 | + "sphinx-doc\_\_sphinx-8269", |\
| 103 | + "sphinx-doc\_\_sphinx-8475", |\
| 104 | + "sphinx-doc\_\_sphinx-8548", |\
| 105 | + "sphinx-doc\_\_sphinx-8551", |\
| 106 | + "sphinx-doc\_\_sphinx-8638", |\
| 107 | + "sphinx-doc\_\_sphinx-8721", |\
| 108 | + "sphinx-doc\_\_sphinx-9229", |\
| 109 | + "sphinx-doc\_\_sphinx-9230", |\
| 110 | + "sphinx-doc\_\_sphinx-9281", |\
| 111 | + "sphinx-doc\_\_sphinx-9320", |\
| 112 | + "sphinx-doc\_\_sphinx-9367", |\
| 113 | + "sphinx-doc\_\_sphinx-9461", |\
| 114 | + "sphinx-doc\_\_sphinx-9698" |\
| 115 | + \], |
| 116 | + "resolved\_ids": \[ |\
| 117 | + "django\_\_django-11815", |\
| 118 | + "django\_\_django-11848", |\
| 119 | + "django\_\_django-11880", |\
| 120 | + "django\_\_django-11951", |\
| 121 | + "django\_\_django-11999", |\
| 122 | + "django\_\_django-12039", |\
| 123 | + "django\_\_django-12050", |\
| 124 | + "django\_\_django-12143", |\
| 125 | + "django\_\_django-12155", |\
| 126 | + "django\_\_django-12193", |\
| 127 | + "django\_\_django-12209", |\
| 128 | + "django\_\_django-12262", |\
| 129 | + "django\_\_django-12276", |\
| 130 | + "django\_\_django-12304", |\
| 131 | + "django\_\_django-12708", |\
| 132 | + "django\_\_django-12713", |\
| 133 | + "django\_\_django-12774", |\
| 134 | + "django\_\_django-9296", |\
| 135 | + "sphinx-doc\_\_sphinx-10466", |\
| 136 | + "sphinx-doc\_\_sphinx-10673", |\
| 137 | + "sphinx-doc\_\_sphinx-7757", |\
| 138 | + "sphinx-doc\_\_sphinx-8035", |\
| 139 | + "sphinx-doc\_\_sphinx-8056", |\
| 140 | + "sphinx-doc\_\_sphinx-8265", |\
| 141 | + "sphinx-doc\_\_sphinx-8269", |\
| 142 | + "sphinx-doc\_\_sphinx-8475", |\
| 143 | + "sphinx-doc\_\_sphinx-8551", |\
| 144 | + "sphinx-doc\_\_sphinx-8721", |\
| 145 | + "sphinx-doc\_\_sphinx-9230", |\
| 146 | + "sphinx-doc\_\_sphinx-9281", |\
| 147 | + "sphinx-doc\_\_sphinx-9320", |\
| 148 | + "sphinx-doc\_\_sphinx-9367", |\
| 149 | + "sphinx-doc\_\_sphinx-9698" |\
| 150 | + \], |
| 151 | + "unresolved\_ids": \[ |\
| 152 | + "django\_\_django-11790", |\
| 153 | + "django\_\_django-11964", |\
| 154 | + "django\_\_django-12273", |\
| 155 | + "django\_\_django-12308", |\
| 156 | + "django\_\_django-12325", |\
| 157 | + "django\_\_django-12406", |\
| 158 | + "sphinx-doc\_\_sphinx-10323", |\
| 159 | + "sphinx-doc\_\_sphinx-10435", |\
| 160 | + "sphinx-doc\_\_sphinx-11510", |\
| 161 | + "sphinx-doc\_\_sphinx-7590", |\
| 162 | + "sphinx-doc\_\_sphinx-7748", |\
| 163 | + "sphinx-doc\_\_sphinx-7985", |\
| 164 | + "sphinx-doc\_\_sphinx-8548", |\
| 165 | + "sphinx-doc\_\_sphinx-8638", |\
| 166 | + "sphinx-doc\_\_sphinx-9461" |\
| 167 | + \], |
| 168 | + "error\_ids": \[\], |
| 169 | + "schema\_version": 2 |
| 170 | +} |

benchmarks/raw/swebench/ornith\_q8\_0\_verified\_mini.jsonADDED
Viewed

|     |     |
| --- | --- |
|  | @@ -0,0 +1,169 @@ |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

|     |     |
| --- | --- |
|  |  |
| 1 | +{ |
| 2 | + "total\_instances": 50, |
| 3 | + "submitted\_instances": 50, |
| 4 | + "completed\_instances": 47, |
| 5 | + "resolved\_instances": 29, |
| 6 | + "unresolved\_instances": 18, |
| 7 | + "empty\_patch\_instances": 3, |
| 8 | + "error\_instances": 0, |
| 9 | + "completed\_ids": \[ |\
| 10 | + "django\_\_django-11790", |\
| 11 | + "django\_\_django-11815", |\
| 12 | + "django\_\_django-11848", |\
| 13 | + "django\_\_django-11880", |\
| 14 | + "django\_\_django-11885", |\
| 15 | + "django\_\_django-11951", |\
| 16 | + "django\_\_django-11964", |\
| 17 | + "django\_\_django-11999", |\
| 18 | + "django\_\_django-12039", |\
| 19 | + "django\_\_django-12050", |\
| 20 | + "django\_\_django-12143", |\
| 21 | + "django\_\_django-12155", |\
| 22 | + "django\_\_django-12193", |\
| 23 | + "django\_\_django-12262", |\
| 24 | + "django\_\_django-12273", |\
| 25 | + "django\_\_django-12276", |\
| 26 | + "django\_\_django-12304", |\
| 27 | + "django\_\_django-12308", |\
| 28 | + "django\_\_django-12325", |\
| 29 | + "django\_\_django-12406", |\
| 30 | + "django\_\_django-12708", |\
| 31 | + "django\_\_django-12713", |\
| 32 | + "django\_\_django-12774", |\
| 33 | + "django\_\_django-9296", |\
| 34 | + "sphinx-doc\_\_sphinx-10435", |\
| 35 | + "sphinx-doc\_\_sphinx-10466", |\
| 36 | + "sphinx-doc\_\_sphinx-10673", |\
| 37 | + "sphinx-doc\_\_sphinx-11510", |\
| 38 | + "sphinx-doc\_\_sphinx-7590", |\
| 39 | + "sphinx-doc\_\_sphinx-7748", |\
| 40 | + "sphinx-doc\_\_sphinx-7757", |\
| 41 | + "sphinx-doc\_\_sphinx-7985", |\
| 42 | + "sphinx-doc\_\_sphinx-8035", |\
| 43 | + "sphinx-doc\_\_sphinx-8056", |\
| 44 | + "sphinx-doc\_\_sphinx-8265", |\
| 45 | + "sphinx-doc\_\_sphinx-8269", |\
| 46 | + "sphinx-doc\_\_sphinx-8475", |\
| 47 | + "sphinx-doc\_\_sphinx-8548", |\
| 48 | + "sphinx-doc\_\_sphinx-8551", |\
| 49 | + "sphinx-doc\_\_sphinx-8638", |\
| 50 | + "sphinx-doc\_\_sphinx-8721", |\
| 51 | + "sphinx-doc\_\_sphinx-9230", |\
| 52 | + "sphinx-doc\_\_sphinx-9281", |\
| 53 | + "sphinx-doc\_\_sphinx-9320", |\
| 54 | + "sphinx-doc\_\_sphinx-9367", |\
| 55 | + "sphinx-doc\_\_sphinx-9461", |\
| 56 | + "sphinx-doc\_\_sphinx-9698" |\
| 57 | + \], |
| 58 | + "incomplete\_ids": \[\], |
| 59 | + "empty\_patch\_ids": \[ |\
| 60 | + "django\_\_django-12209", |\
| 61 | + "sphinx-doc\_\_sphinx-10323", |\
| 62 | + "sphinx-doc\_\_sphinx-9229" |\
| 63 | + \], |
| 64 | + "submitted\_ids": \[ |\
| 65 | + "django\_\_django-11790", |\
| 66 | + "django\_\_django-11815", |\
| 67 | + "django\_\_django-11848", |\
| 68 | + "django\_\_django-11880", |\
| 69 | + "django\_\_django-11885", |\
| 70 | + "django\_\_django-11951", |\
| 71 | + "django\_\_django-11964", |\
| 72 | + "django\_\_django-11999", |\
| 73 | + "django\_\_django-12039", |\
| 74 | + "django\_\_django-12050", |\
| 75 | + "django\_\_django-12143", |\
| 76 | + "django\_\_django-12155", |\
| 77 | + "django\_\_django-12193", |\
| 78 | + "django\_\_django-12209", |\
| 79 | + "django\_\_django-12262", |\
| 80 | + "django\_\_django-12273", |\
| 81 | + "django\_\_django-12276", |\
| 82 | + "django\_\_django-12304", |\
| 83 | + "django\_\_django-12308", |\
| 84 | + "django\_\_django-12325", |\
| 85 | + "django\_\_django-12406", |\
| 86 | + "django\_\_django-12708", |\
| 87 | + "django\_\_django-12713", |\
| 88 | + "django\_\_django-12774", |\
| 89 | + "django\_\_django-9296", |\
| 90 | + "sphinx-doc\_\_sphinx-10323", |\
| 91 | + "sphinx-doc\_\_sphinx-10435", |\
| 92 | + "sphinx-doc\_\_sphinx-10466", |\
| 93 | + "sphinx-doc\_\_sphinx-10673", |\
| 94 | + "sphinx-doc\_\_sphinx-11510", |\
| 95 | + "sphinx-doc\_\_sphinx-7590", |\
| 96 | + "sphinx-doc\_\_sphinx-7748", |\
| 97 | + "sphinx-doc\_\_sphinx-7757", |\
| 98 | + "sphinx-doc\_\_sphinx-7985", |\
| 99 | + "sphinx-doc\_\_sphinx-8035", |\
| 100 | + "sphinx-doc\_\_sphinx-8056", |\
| 101 | + "sphinx-doc\_\_sphinx-8265", |\
| 102 | + "sphinx-doc\_\_sphinx-8269", |\
| 103 | + "sphinx-doc\_\_sphinx-8475", |\
| 104 | + "sphinx-doc\_\_sphinx-8548", |\
| 105 | + "sphinx-doc\_\_sphinx-8551", |\
| 106 | + "sphinx-doc\_\_sphinx-8638", |\
| 107 | + "sphinx-doc\_\_sphinx-8721", |\
| 108 | + "sphinx-doc\_\_sphinx-9229", |\
| 109 | + "sphinx-doc\_\_sphinx-9230", |\
| 110 | + "sphinx-doc\_\_sphinx-9281", |\
| 111 | + "sphinx-doc\_\_sphinx-9320", |\
| 112 | + "sphinx-doc\_\_sphinx-9367", |\
| 113 | + "sphinx-doc\_\_sphinx-9461", |\
| 114 | + "sphinx-doc\_\_sphinx-9698" |\
| 115 | + \], |
| 116 | + "resolved\_ids": \[ |\
| 117 | + "django\_\_django-11815", |\
| 118 | + "django\_\_django-11848", |\
| 119 | + "django\_\_django-11880", |\
| 120 | + "django\_\_django-11951", |\
| 121 | + "django\_\_django-11999", |\
| 122 | + "django\_\_django-12039", |\
| 123 | + "django\_\_django-12050", |\
| 124 | + "django\_\_django-12143", |\
| 125 | + "django\_\_django-12155", |\
| 126 | + "django\_\_django-12262", |\
| 127 | + "django\_\_django-12276", |\
| 128 | + "django\_\_django-12304", |\
| 129 | + "django\_\_django-12708", |\
| 130 | + "django\_\_django-12713", |\
| 131 | + "django\_\_django-12774", |\
| 132 | + "django\_\_django-9296", |\
| 133 | + "sphinx-doc\_\_sphinx-10466", |\
| 134 | + "sphinx-doc\_\_sphinx-7757", |\
| 135 | + "sphinx-doc\_\_sphinx-7985", |\
| 136 | + "sphinx-doc\_\_sphinx-8035", |\
| 137 | + "sphinx-doc\_\_sphinx-8265", |\
| 138 | + "sphinx-doc\_\_sphinx-8269", |\
| 139 | + "sphinx-doc\_\_sphinx-8475", |\
| 140 | + "sphinx-doc\_\_sphinx-8551", |\
| 141 | + "sphinx-doc\_\_sphinx-8721", |\
| 142 | + "sphinx-doc\_\_sphinx-9281", |\
| 143 | + "sphinx-doc\_\_sphinx-9320", |\
| 144 | + "sphinx-doc\_\_sphinx-9367", |\
| 145 | + "sphinx-doc\_\_sphinx-9698" |\
| 146 | + \], |
| 147 | + "unresolved\_ids": \[ |\
| 148 | + "django\_\_django-11790", |\
| 149 | + "django\_\_django-11885", |\
| 150 | + "django\_\_django-11964", |\
| 151 | + "django\_\_django-12193", |\
| 152 | + "django\_\_django-12273", |\
| 153 | + "django\_\_django-12308", |\
| 154 | + "django\_\_django-12325", |\
| 155 | + "django\_\_django-12406", |\
| 156 | + "sphinx-doc\_\_sphinx-10435", |\
| 157 | + "sphinx-doc\_\_sphinx-10673", |\
| 158 | + "sphinx-doc\_\_sphinx-11510", |\
| 159 | + "sphinx-doc\_\_sphinx-7590", |\
| 160 | + "sphinx-doc\_\_sphinx-7748", |\
| 161 | + "sphinx-doc\_\_sphinx-8056", |\
| 162 | + "sphinx-doc\_\_sphinx-8548", |\
| 163 | + "sphinx-doc\_\_sphinx-8638", |\
| 164 | + "sphinx-doc\_\_sphinx-9230", |\
| 165 | + "sphinx-doc\_\_sphinx-9461" |\
| 166 | + \], |
| 167 | + "error\_ids": \[\], |
| 168 | + "schema\_version": 2 |
| 169 | +} |

benchmarks/swebench-agent-evals.mdADDED
Viewed

|     |     |
| --- | --- |
|  | @@ -0,0 +1,75 @@ |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

|     |     |
| --- | --- |
|  |  |
| 1 | +\# SWE-bench Agent Evaluations |
| 2 | + |
| 3 | +Date: 2026-06-29 |
| 4 | + |
| 5 | +Harness: mini-SWE-agent plus the SWE-bench evaluation harness from |
| 6 | +\`/home/ripper/dsv4-local/eval/swebench\`. |
| 7 | + |
| 8 | +\## Full SWE-bench Verified 100-task Slice |
| 9 | + |
| 10 | +This is the faithful 100-task run requested for reasoning mode. The run used |
| 11 | +\`SWE-bench/SWE-bench\_Verified\`, split \`test\`, slice \`0:100\`. |
| 12 | + |
| 13 | +\| Profile \| Reasoning \| Temperature \| Resolved \| Score \| |
| 14 | +\|---\|---\|---:\|---:\|---:\| |
| 15 | +\| Q6\_K GGUF \| on \| 1.0 \| 69/100 \| 69.0% \| |
| 16 | + |
| 17 | +Score chart: |
| 18 | + |
| 19 | +\| Profile \| Resolved chart \| |
| 20 | +\|---\|---\| |
| 21 | +\| Q6\_K reasoning-on \| \`##################################################################### 69.0%\` \| |
| 22 | + |
| 23 | +Notes: |
| 24 | + |
| 25 | +\- \`enable\_thinking=true\`, \`preserve\_thinking=true\`. |
| 26 | +\- No explicit thinking budget was enforced. |
| 27 | +\- \`top\_p=1.0\`, \`max\_output\_tokens=8192\`. |
| 28 | +\- 94/100 tasks submitted patches. |
| 29 | +\- 6/100 ended as empty patch or timeout. |
| 30 | +\- Rerunning the two original infrastructure-error cases produced 0 remaining |
| 31 | + infra errors. |
| 32 | +\- The original harness wrapper row can look like \`69/500 = 13.80%\` because the |
| 33 | + first invocation did not pass the intended instance filters. The corrected |
| 34 | + intended score for the slice is \`69/100 = 69.0%\`. |
| 35 | + |
| 36 | +Raw summary: |
| 37 | +\[\`benchmarks/raw/swebench/ornith-q6-think-verified100-summary.json\`\](raw/swebench/ornith-q6-think-verified100-summary.json). |
| 38 | + |
| 39 | +\## SWE-bench Verified Mini Quant Sweep |
| 40 | + |
| 41 | +This sweep used \`MariusHobbhahn/swe-bench-verified-mini\`, split \`test\`, with the |
| 42 | +same mini-SWE-agent setup across quants. These are 50-task verified-mini scores, |
| 43 | +not the full 100-task slice above. |
| 44 | + |
| 45 | +\| Quant/profile \| Resolved \| Score \| Completed \| Empty patch \| Errors \| |
| 46 | +\|---\|---:\|---:\|---:\|---:\|---:\| |
| 47 | +\| Q6\_K \| 33/50 \| 66.0% \| 48/50 \| 2 \| 0 \| |
| 48 | +\| Q3\_K\_M \| 32/50 \| 64.0% \| 47/50 \| 3 \| 0 \| |
| 49 | +\| Q5\_K\_M \| 32/50 \| 64.0% \| 49/50 \| 1 \| 0 \| |
| 50 | +\| Q4\_K\_M \| 31/50 \| 62.0% \| 48/50 \| 2 \| 0 \| |
| 51 | +\| IQ4\_XS \| 30/50 \| 60.0% \| 45/50 \| 5 \| 0 \| |
| 52 | +\| IQ4\_XS-MTP-graft-headQ6 \| 30/50 \| 60.0% \| 48/50 \| 2 \| 0 \| |
| 53 | +\| Q8\_0 \| 29/50 \| 58.0% \| 47/50 \| 3 \| 0 \| |
| 54 | + |
| 55 | +Resolved chart: |
| 56 | + |
| 57 | +\| Quant/profile \| Resolved chart \| |
| 58 | +\|---\|---\| |
| 59 | +\| Q6\_K \| \`################################################################## 66.0%\` \| |
| 60 | +\| Q3\_K\_M \| \`################################################################ 64.0%\` \| |
| 61 | +\| Q5\_K\_M \| \`################################################################ 64.0%\` \| |
| 62 | +\| Q4\_K\_M \| \`############################################################## 62.0%\` \| |
| 63 | +\| IQ4\_XS \| \`############################################################ 60.0%\` \| |
| 64 | +\| IQ4\_XS-MTP-graft-headQ6 \| \`############################################################ 60.0%\` \| |
| 65 | +\| Q8\_0 \| \`########################################################## 58.0%\` \| |
| 66 | + |
| 67 | +Raw reports: |
| 68 | + |
| 69 | +\- \[\`ornith\_q6\_k\_verified\_mini.json\`\](raw/swebench/ornith\_q6\_k\_verified\_mini.json) |
| 70 | +\- \[\`ornith\_q3\_k\_m\_verified\_mini.json\`\](raw/swebench/ornith\_q3\_k\_m\_verified\_mini.json) |
| 71 | +\- \[\`ornith\_q5\_k\_m\_verified\_mini.json\`\](raw/swebench/ornith\_q5\_k\_m\_verified\_mini.json) |
| 72 | +\- \[\`ornith\_q4\_k\_m\_verified\_mini.json\`\](raw/swebench/ornith\_q4\_k\_m\_verified\_mini.json) |
| 73 | +\- \[\`ornith\_iq4\_xs\_verified\_mini.json\`\](raw/swebench/ornith\_iq4\_xs\_verified\_mini.json) |
| 74 | +\- \[\`ornith\_iq4\_xs\_mtp\_graft\_verified\_mini.json\`\](raw/swebench/ornith\_iq4\_xs\_mtp\_graft\_verified\_mini.json) |
| 75 | +\- \[\`ornith\_q8\_0\_verified\_mini.json\`\](raw/swebench/ornith\_q8\_0\_verified\_mini.json) |

Unwrap lines

StripeM-Inner