# LordNeel Ornith-1.0-35B GGUF Quality Ladder

Source: [LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1](https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1)

Single-GPU llama.cpp GGUF package with measured fidelity against BF16 reference.

## Fidelity vs BF16 (top-64 KLD, 32 coding prompts)

Lower KLD = closer to BF16 reference. Measured at next-token level, temperature=-1.

| Artifact | File Size | Mean KLD (nats) | Greedy top-1 | MTP? |
|---|---|---|---|---|
| Q8_0 | 36.9 GB | 0.0108 | 31/32 | ❌ |
| Q6_K | 28.5 GB | 0.0165 | 32/32 | ❌ |
| **Q5_K_M** | **24.7 GB** | **0.0354** | **30/32** | ❌ |
| **IQ4_XS-MTP-graft** | **19.6 GB** | **0.0731** | **29/32** | ✅ ~1.35× |
| Q4_K_M | 21.2 GB | 0.0864 | 29/32 | ❌ |
| IQ4_XS (plain) | 18.9 GB | 0.1426 | 27/32 | ❌ |
| Q3_K_M | 16.8 GB | 0.3620 | 27/32 | ❌ |

Key find: **IQ4_XS-MTP-graft has BETTER fidelity than Q4_K_M** (0.0731 vs 0.0864) despite being 1.6 GB smaller. The MTP graft head (Q6_K donor with cross-model refinement) improves the base model's next-token distribution.

## IQ4_XS-MTP-graft Details

- **Architecture:** Qwen3.5-35B-A3B MoE (qwen35moe, 40 blocks)
- **MTP type:** Integrated graft — native `--spec-type draft-mtp` self-speculation
- **Acceptance rate:** 82.01% at `--spec-draft-n-max 2`
- **Speedup:** ~1.35× single-stream decode (233.8 vs 172.6 t/s client)
- **KV overhead:** ~280 MiB extra for MTP draft context
- **Greedy lossless?** Next-token KLD vs target-only = 0.0 (32/32 identical). Long sequences: 93.4% token-position match over 512 tokens.

## Throughput (short context, single stream)

| Quant | c1 tok/s | c4 tok/s | c16 tok/s |
|---|---|---|---|
| Q5_K_M | 236.7 | 311.0 | 638.6 |
| Q4_K_M | 243.3 | 458.3 | 655.6 |
| IQ4_XS-MTP | 319.5 (active) | — | — |

Note: These benchmark from LordNeel's RTX 4090. On RTX 2080 Ti (sm75) with `--n-cpu-moe` offloading, expect ~40-50% of these numbers.

## For RTX 2080 Ti 11GB (Our Setup)

These file sizes are irrelevant for VRAM fitting — MoE offloading (`--n-cpu-moe`) keeps expert weights on CPU. Only non-expert layers (~2-3 GB) need GPU regardless of quantization.

**Estimated speed with `--n-cpu-moe 32` on sm75:**
| Quant | Est. t/s | MTP? | Quality rank |
|---|---|---|---|
| APEX-I-Compact (17 GB, current) | 28.5 | ✅ MTP | Worst (below Q4) |
| IQ4_XS-MTP-graft (19.6 GB) | ~35-40 | ✅ MTP | Better than Q4_K_M |
| Q4_K_M (21.2 GB) | ~35-40 | ❌ | Good baseline |
| Q5_K_M (24.7 GB) | ~30-35 | ❌ | Best available |