# Eagle3 Speculator Setup

## Converting safetensors → GGUF

```bash
# 1. Download base model config (for vocabulary/tokenizer)
pip install huggingface_hub
python -c "
from huggingface_hub import snapshot_download
snapshot_download('Qwen/Qwen3-30B-A3B-Thinking-2507',
    local_dir='/tmp/base-model-config',
    allow_patterns=['tokenizer.json', 'tokenizer_config.json', 'config.json'])
"

# 2. Convert speculator (requires torch in venv)
/home/rurouni/.hermes/hermes-agent/venv/bin/python \
  /home/rurouni/src/llama.cpp-master/convert_hf_to_gguf.py \
  /path/to/eagle3-speculator-dir \
  --outfile /path/to/output.gguf \
  --target-model-dir /tmp/base-model-config
```

Requires `torch` (available in Hermes venv). The `--target-model-dir` provides architecture metadata (vocabulary, hidden size, etc.) that the speculator GGUF needs to pair with its base model.

## Pre-converted GGUFs

| Base Model | Speculator | Status |
|-----------|-----------|--------|
| Qwen3-30B-A3B-Thinking-2507 | `RedHatAI/Qwen3-30B-A3B-Thinking-2507-speculator.eagle3` | ✅ Works with our b9743 binary |
| Qwen3-Coder-Next (80B) | `jdluzen/Qwen3-Coder-Next-Eagle3-GGUF` | ❌ "Not available in mainline llama.cpp yet" — qwen3next arch Eagle3 not supported |

## Usage

```bash
llama-server \
  --model <base-model.gguf> \
  --spec-draft-model <eagle3.gguf> \
  --spec-type draft-eagle3 \
  --spec-draft-n-max 5 \
  --spec-draft-p-min 0.75 \
  --spec-draft-type-k q8_0 \
  --spec-draft-type-v q8_0
```

## Key Details

- **Architecture-locked:** Eagle3 speculators are trained for one specific base model. They will NOT work with any other model.
- **n_max=5 recommended:** The RedHatAI speculator was trained at k=5 (draft length). Using lower values limits acceptance.
- **CPU-offload bottleneck:** When using `--n-cpu-moe`, the main model's expert CPU evaluation is the bottleneck, not token prediction speed. Eagle3 speedup is marginal (2-3%) in this configuration.
- **Coder-Next Eagle3:** Pre-built GGUF exists from `jdluzen/Qwen3-Coder-Next-Eagle3-GGUF` but DOES NOT WORK with our b9743 binary. The `qwen3next` arch's Eagle3 support isn't in mainline llama.cpp yet. Check for newer builds if this changes.
- **Conversion gotcha:** The speculator model card says "not available in mainline llama.cpp yet" — always verify compatibility before downloading large Eagle3 models.

## Tuning Observations (RTX 2080 Ti)

- n_max=3 vs n_max=5: virtually identical speed (+0.4 t/s difference)
- The bottleneck is expert CPU evaluation via `--n-cpu-moe`, not draft prediction speed
- On a GPU-bound setup (all layers on GPU, no CPU offload), Eagle3 would provide meaningful gains (estimated 30-50% based on vLLM benchmarks on 8× GPU hardware)
- 30B-Thinking + Eagle3: 35.9 t/s vs 34.8 t/s without spec decoding (+3%)