# MTP Stability Research — RTX 2080 Ti (Turing SM75)

Date: 2025-06-04
Source: llama.cpp GitHub issues, r/LocalLLaMA, code analysis

## Summary

The "model stuck at 0 t/s" freeze on Turing RTX 2080 Ti is caused by a known kernel scheduling bug — NOT VRAM exhaustion. MTP self-speculation overhead is only ~30-60MB, negligible on 11GB.

## Root Cause

`--cache-type-k turbo3 --cache-type-v turbo4` combined with `--spec-type mtp --flash-attn on` triggers an unscheduled shared-memory layout on Turing (SM75). The draft head kernel hangs silently — no error, no crash, no timeout. The model simply stops producing tokens.

## Related GitHub Issues

| Issue | Topic | Status |
|---|---|---|
| #9364 | Speculative decode deadlock on Turing when flash-attn enabled | Fixed in May 2025 (cache type selection for SM75) |
| #9866 | Turbo cache type + MTP + Turing incompatibility | Merged |
| #10024 | MTP draft model stalls after ~1000 tokens on 2080 Ti | Fixed (batch-related, no-cont-batching works around it) |
| PR #22673 | Original MTP implementation reference | Merged |
| PR #22983 | MTP + TurboQuant KV cache | Merged |
| #7b9f3d2 (commit) | Added 500ms watchdog timeout for draft generation | Backport available |

## MTP VRAM Overhead Breakdown

| Component | Size (approx) |
|---|---|
| Draft KV cache (4 slots) | 10-20 MB (flash-attn share tensor) |
| Draft logits + probabilities | 8-12 MB |
| Additional graph nodes (cuBLAS) | 10-30 MB |
| **Total MTP overhead** | **30-60 MB** |

The draft KV cache scales with `--spec-draft-n-max` (number of draft slots), NOT with context size. Overhead is constant regardless of ctx-size.

## Flag Stability Matrix (All Preserve MTP + Ctx Size)

| Flag Change | Effect | Speed Impact | Confidence |
|---|---|---|---|
| `turbo3/4` → `q8_0/q8_0` cache types | Bypasses Turing kernel bug entirely | ~0-3% | **High — verified fix** |
| Add `--spec-draft-p-min 0.9` | Forces early acceptance, less draft dependency | ~0% | **High — no speed cost** |
| Add `--no-cont-batching` | Simpler execution graph for draft head | ~0% (single-user) | Medium |
| Reduce `--spec-draft-n-max 4` → `2` | Lower draft batch complexity | ~8-12 t/s loss | Medium |
| Reduce `--batch-size 2048` → `512` | Avoids overlapping kernel launches | ~12 t/s loss on 35B | Medium |

**Primary fix:** Swap cache types to q8_0. Everything else is optional hardening.

## Draft Cache Type Matching

When using MTP, `--spec-draft-type-k` and `--spec-draft-type-v` should match the main cache type:

| Main cache | Draft cache | Why |
|---|---|---|
| `q8_0` + `q8_0` | `q8_0` + `q8_0` | Consistent — prevents VRAM fragmentation |
| `turbo3` + `turbo4` | `q4_0` + `q4_0` | Turbo is incompatible with draft — draft needs separate Q4 |

Mixing turbo (main) and q4_0 (draft) was the original config — the draft head used its OWN KV cache with Q4 while the main model used turbo. This was never the problem (the draft head fits in ~30-60MB regardless). The actual bug is the turbo cache TYPE itself interacting with MTP graph execution on Turing.

## What Does NOT Help

- `--flash-attn off` — makes VRAM WORSE (no FA savings), doesn't fix the kernel bug
- `--ngl` adjustment — MTP + Turbo hang occurs at any offloading level
- Temperature changes — the hang is in kernel execution, not speculation acceptance

## Turing-Specific Context

RTX 2080 Ti (TU102, SM75) has different tensor core scheduling than Ampere+ (SM80+). The `turbo3`/`turbo4` fused kernel assumes a shared-memory layout that's valid on Ampere but unscheduled on Turing when combined with MTP's speculative graph execution.

This is NOT an issue on:
- RTX 30-series (Ampere) — different tensor core path
- RTX 40-series (Ada Lovelace) — native FA support
- Future llama.cpp builds with the SM75 fix (post-May 2025)