# TurboQuant KV Cache Compression

## Overview

Google Research's KV cache compression (ICLR 2026). Compresses key-value cache using polar coordinates + 1-bit error correction.

**Research blog:** https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/

## Cache Types

| Type | Ratio | Notes |
|------|-------|-------|
| `turbo2` | ~2x | Lightweight, best quality |
| `turbo3` | ~3x | Balanced |
| `turbo4` | ~4x | Most aggressive |

## Recommended Configuration

```
-ctk turbo4 -ctv turbo2
```

TurboQuant keys + standard turbo2 values gives the best compression/quality tradeoff.

## Current Status

**BoFan's fork is now installed as the default `llama-server`** (as of May 12, 2026). Build recipe:

```bash
git clone -b merge-mtp-turboquant --depth 1 \
  https://github.com/BoFan-tunning/llama.cpp-MTP-TurboQuant.git /tmp/llama-tq
cd /tmp/llama-tq && mkdir -p build && cd build

CUDAHOSTCXX=/usr/bin/gcc-13 cmake .. \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DCUDAToolkit_ROOT=/usr/local/cuda \
  -DBUILD_SHARED_LIBS=OFF     # ⚠️ Static binary, not dynamic

cmake --build . --target llama-server --config Release -j$(nproc)
sudo cp bin/llama-server /usr/local/bin/llama-server
```

**Key flags:**
- `-DBUILD_SHARED_LIBS=OFF` — produces a 100+ MB static binary (without this, you get a 9 MB dynamic binary that breaks if build tree is cleaned)
- `CUDAHOSTCXX=/usr/bin/gcc-13` — bypasses CUDA 12.x + GCC 14 incompatibility without needing an nvcc wrapper script

## Fork Status

### Primary (Debian Hermes host, RTX 2080 Ti): EsmaeelNabil/llama.cpp-mtp-turbo-quant

- **https://github.com/EsmaeelNabil/llama.cpp-mtp-turbo-quant** (branch `feat/mtp-turboquant-kv-cache`)
- ⭐9, last pushed May 12, 2026
- **All-in-one merge** of am17an MTP + TheTom TurboQuant — conflict-resolved by Claude Sonnet
- Based on the **newest upstream** (May 12, 2026, commit `a578bb6b`) — picks up mtmd modality fixes, gpt-oss WebGPU, MiMo v2.5 vision
- **Turbo2/3/4 KV cache types** (`--cache-type-k turbo4 --cache-type-v turbo2`) natively supported
- **CUDA FA kernels for all turbo combinations:** turbo3↔q8_0, turbo3↔f16, turbo3↔turbo2, turbo3↔turbo4, and all permutations
- **MTP speculative decoding** (`--spec-type mtp`) with `--spec-draft-n-max`
- Head-dim exclusion merged from both MTP (192) and TurboQuant (640) — handles wide models correctly
- **GCC 13 required** for CUDA 12.6 host compilation; set `CUDAHOSTCXX=/usr/bin/g++-13` + `CMAKE_CUDA_COMPILER` + `CMAKE_CUDA_HOST_COMPILER`

Build recipe:
```bash
git clone https://github.com/EsmaeelNabil/llama.cpp-mtp-turbo-quant.git /tmp/llama-mtp-tq
cd /tmp/llama-mtp-tq && git checkout feat/mtp-turboquant-kv-cache
mkdir -p build && cd build

CUDAHOSTCXX=/usr/bin/g++-13 cmake .. \
  -DCMAKE_C_COMPILER=/usr/bin/gcc-13 \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++-13 \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.6/bin/nvcc \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-13 \
  -DCMAKE_CUDA_ARCHITECTURES=75 \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA=ON \
  -DGGML_CUDA_GRAPHS=ON \
  -DGGML_CPU=ON \
  -DGGML_OPENMP=ON \
  -DGGML_NATIVE=ON \
  -DGGML_LLAMAFILE=ON \
  -DGGML_BUILD_EXAMPLES=OFF \
  -DGGML_BUILD_TESTS=OFF \
  -DCMAKE_BUILD_TYPE=Release

make -j$(nproc) llama-server
sudo cp bin/llama-server /usr/local/bin/llama-server
```

### Secondary (Ray's CachyOS, GTX 1080 Ti): BoFan-tunning/llama.cpp-MTP-TurboQuant

- **https://github.com/BoFan-tunning/llama.cpp-MTP-TurboQuant** (branch `merge-mtp-turboquant`)
- ⭐8, updated May 12, 2026
- Based on upstream llama.cpp from **~May 4, 2026** (modern enough for Gemma 3/4, Qwen3.6 vision)
- **MTP + TurboQuant fused** — dedicated Qwen3.6-27B MTP support
- **Crash fixes included:**
  - Multimodal crash fix (seeds MTP drafts from logical positions)
  - `turbo2/3/4` whitelist for `-ctk/-ctv` (fails without this on some model arches)
  - FATTN `turbo*` + F16 template instantiations (no runtime dispatch failures)
- **No RDNA2 baggage** — clean CUDA backend, no SWA hang
- Upstream base has all model architectures (Gemma 3/4, Phi-4, Qwen3.6 MTP, etc.)

**Verified models (GTX 1080 Ti, May 12, 2026):**
| Model | TurboQuant | Status |
|-------|-----------|--------|
| Phi-4 14B (Q4_K) | `-ctk turbo4 -ctv turbo2` | ✅ Works — correct inference |
| Gemma 3 12B (Q4_K_M) | `-ctk turbo4 -ctv turbo2` | ✅ Works — no SWA hang, correct inference |

### Secondary: Stormrage34/llama.cpp-turboquant-hip

- https://github.com/Stormrage34/llama.cpp-turboquant-hip
- Actively maintained (May 2026, v0.3.1 stabilized)
- Includes TriAttention, RDNA2/3 CUDA optimizations, GPU idle temp display
- ⚠️ **HEAD breaks SWA models** (Gemma 3/4 hang at `sched_reserve`) due to RDNA2 flash attention changes
- ⚠️ **All commits, including pre-RDNA2 `ba62fc7`, crash Gemma 3** with SIGSEGV (exit code 139) because the fork's base upstream is too old for Gemma 3 support
- Use only for non-SWA models (Phi-4, Qwen) when BoFan fork doesn't apply

### Comparison

| Feature | EsmaeelNabil (2080 Ti) | BoFan (1080 Ti) | Stormrage HEAD | Stormrage ba62fc7 |
|---------|------------------------|-------------------|---------------|-------------------|
| Upstream base | ~May 12, 2026 | ~May 4, 2026 | ~Apr 23, 2026 | ~Apr 23, 2026 |
| TurboQuant | ✅ turbo2/3/4 | ✅ | ✅ | ✅ |
| MTP support | ✅ (fused) | ✅ (fused) | ✅ | ✅ |
| CUDA FA for turbo | ✅ All combos | ✅ | ✅ | ❌ (missing) |
| **Gemma 3 SWA** | ✅ Should work | ✅ Verified | ❌ Hangs | ❌ Crashes (SIGSEGV) |
| Gemma 4 SWA | ✅ Should work | ✅ Should work | ❌ Hangs | ❌ Likely crashes |
| Phi-4 14B | ✅ Should work | ✅ Verified | ✅ (117 t/s prompt) | ✅ (117 t/s prompt) |
| RDNA2 opt | ❌ (none) | ❌ (none) | ✅ (AMD) | ❌ (none) |
| Multimodal fix | ✅ (mtmd from upstream) | ✅ | ❌ | ❌ |
| sm_75 tensor cores | ✅ (compile with arch=75) | N/A (sm_61 target) | N/A | N/A |
| Head-dim merge | ✅ 192+640 exclusion | ❌ (not merged) | ❌ | ❌ |

## Building — EsmaeelNabil Fork (Debian Hermes host, RTX 2080 Ti, sm_75)

This is the canonical build for the Debian Hermes host. The `feat/mtp-turboquant-kv-cache` branch is the "all-in-one" merge.

```bash
# Clone (full clone — single branch shallow clone may miss merge-base)
git clone https://github.com/EsmaeelNabil/llama.cpp-mtp-turbo-quant.git ~/src/llama.cpp-mtp-turbo
cd ~/src/llama.cpp-mtp-turbo && git checkout feat/mtp-turboquant-kv-cache
mkdir -p build-cuda-sm75 && cd build-cuda-sm75

# Configure — CUDA 12.6 with nvcc not on PATH
CUDAHOSTCXX=/usr/bin/g++-13 cmake .. \
  -DCMAKE_C_COMPILER=/usr/bin/gcc-13 \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++-13 \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.6/bin/nvcc \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-13 \
  -DCMAKE_CUDA_ARCHITECTURES=75 \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA=ON \
  -DGGML_CUDA_GRAPHS=ON \
  -DGGML_CPU=ON \
  -DGGML_OPENMP=ON \
  -DGGML_NATIVE=ON \
  -DGGML_LLAMAFILE=ON \
  -DGGML_BUILD_EXAMPLES=OFF \
  -DGGML_BUILD_TESTS=OFF \
  -DGGML_CCACHE=OFF \
  -DCMAKE_BUILD_TYPE=Release

# Build only the server target
make -j$(nproc) llama-server

# Install (backup existing 1080 Ti binaries first)
sudo cp /usr/local/bin/llama-server /usr/local/bin/llama-server-sm61.bak
sudo cp bin/llama-server /usr/local/bin/llama-server

# Verify
llama-server --version
# Should show: compute capability 7.5, Total VRAM: 11002 MiB
```

**MTP flags** (available in this build):
```
--spec-type mtp              # Enable multi-token prediction speculative decoding
--spec-draft-n-max N         # Max draft tokens per step
--cache-type-k turbo4        # TurboQuant key cache (4-bit)
--cache-type-v turbo2        # TurboQuant value cache (2-bit)
```

## Building — BoFan Fork (Ray's CachyOS, GTX 1080 Ti, sm_61)

This is now the canonical build. The resulting binary replaces `/usr/local/bin/llama-server`.

```bash
# Clone (shallow)
git clone -b merge-mtp-turboquant --depth 1 \
  https://github.com/BoFan-tunning/llama.cpp-MTP-TurboQuant.git /tmp/llama-tq
cd /tmp/llama-tq && mkdir -p build && cd build

# Configure with static build
CUDAHOSTCXX=/usr/bin/gcc-13 cmake .. \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DCUDAToolkit_ROOT=/usr/local/cuda \
  -DBUILD_SHARED_LIBS=OFF

# Build llama-server target
cmake --build . --target llama-server --config Release -j$(nproc)

# Verify static
ls -lh bin/llama-server   # Should be 100+ MB
ldd bin/llama-server      # Should show NO llama/ggml libs

# Install (keep backup)
sudo cp /usr/local/bin/llama-server /usr/local/bin/llama-server.bak
sudo cp bin/llama-server /usr/local/bin/llama-server

# Cleanup temp
rm -rf /tmp/llama-tq
```

**The `BUILD_SHARED_LIBS=OFF` is critical.** Without it, cmake may default `BUILD_SHARED_LIBS=ON` on some platforms, producing a 9 MB dynamically-linked binary that depends on `.so` files from the build tree. If `/tmp/llama-tq/build/bin/` gets cleaned up later, the installed binary silently fails at runtime.

## Building — Stormrage Fork (Legacy, Fallback Only)

Use only if BoFan doesn't work for your scenario.

### Stormrage Latest HEAD

```bash
git clone https://github.com/Stormrage34/llama.cpp-turboquant-hip /tmp/stormrage
cd /tmp/stormrage && mkdir -p build && cd build
CUDAHOSTCXX=/usr/bin/gcc-13 cmake .. \
  -DGGML_CUDA=ON -DGGML_CUDA_FA=ON -DCMAKE_BUILD_TYPE=Release \
  -DCUDAToolkit_ROOT=/usr/local/cuda
cmake --build . --target llama-server --config Release -j$(nproc)
sudo cp bin/llama-server /usr/local/bin/llama-server-tq
```

Works for non-SWA models (Phi-4: 117 t/s prompt). **Hangs with Gemma 3/4.**

### Stormrage Pre-RDNA2 (ba62fc7 — Does NOT Fix Gemma)

While this commit removes the RDNA2 flash attention changes, the base upstream code is still too old for Gemma 3/4. The server **crashes with SIGSEGV** (exit code 139) at `sched_reserve`, not hangs. Best use case: Phi-4 / Qwen on systems where BoFan's fork won't build.

```bash
git clone https://github.com/Stormrage34/llama.cpp-turboquant-hip /tmp/stormrage
cd /tmp/stormrage && git checkout ba62fc7
mkdir -p build && cd build
CUDAHOSTCXX=/usr/bin/gcc-13 cmake .. \
  -DGGML_CUDA=ON -DGGML_CUDA_FA=ON -DCMAKE_BUILD_TYPE=Release \
  -DCUDAToolkit_ROOT=/usr/local/cuda
cmake --build . --target llama-server --config Release -j$(nproc)
sudo cp bin/llama-server /usr/local/bin/llama-server-tq
```

## Known Issues

- `turbo4` needs `SET_ROWS` GPU operation — crashes on Vulkan backends that don't support it
- CUDA backend is reported to work by the community
- GTX 1080 Ti (CC 6.1) cannot use `--fa` flag — Flash Attention requires CC 7.0+. Build with `-DGGML_CUDA_FA=OFF` if needed (the fork defaults it ON).

### SWA Incompatibility (Stormrage Fork Only)

The Stormrage fork **hangs at `sched_reserve: reserving ...`** for models with sliding window attention (SWA):

| Model | SWA? | BoFan Fork | Stormrage Fork |
|-------|------|-----------|---------------|
| Phi-4 14B | ❌ | ✅ | ✅ (117 t/s prompt) |
| Qwen3.6-27B | ❌ | ✅ | ✅ (partial offload) |
| Gemma 3 12B | ✅ | ✅ | ❌ Hangs |
| Gemma 4 26B | ✅ | ✅ should work | ❌ Hangs |
| Llama 3.x | ❌ | ✅ | ✅ |

**Does NOT affect the BoFan fork.** The hanging was caused by RDNA2 flash attention changes in the Stormrage fork's scheduler. BoFan's fork has a clean CUDA backend.