# Benchmarking New GGUFs: Download → Verify → Speed Test

Full workflow for downloading, verifying, and speed-testing one or more new GGUF model files using `curl` + `llama-server` log parsing.

## 1. Download GGUFs from HuggingFace

### Preferred method: `curl -sL` (fast, no auth needed)

```bash
curl -sL -o /models/llm/speedtest/ModelName-Q4_K_M.gguf \
  "https://huggingface.co/org/ModelName-GGUF/resolve/main/ModelName-Q4_K_M.gguf"
```

Key points:
- **`-L` is mandatory** — HuggingFace serves a 302 redirect to a CDN. Without `-L`, you get a tiny HTML redirect page.
- **No HF_TOKEN needed** — 95 MB/s achieved without auth on this connection. Auth would get higher rate limits but is not required.
- **`-s` for silent progress** — no progress bar in output.
- Use `-o <path>` to write to a specific location.

### Fallback: `hf download` CLI (requires auth)

```bash
hf download org/ModelName-GGUF ModelName-Q4_K_M.gguf --quiet
```

With no HF_TOKEN, this is heavily throttled (~8 KB/s). Only use if you have a valid HF token set.

### Parallel downloads: pitfall with truncation

Downloading multiple GGUFs in parallel with `curl -sL` works, but all downloads may stop simultaneously when the bandwidth pool adjusts. In practice, with 5 parallel 7-20 GB downloads, all five stopped at ~15 GB simultaneously, leaving large models truncated.

**Fix:** Resume truncated downloads with `curl -C -`:

```bash
curl -sL -C - -o /models/llm/speedtest/ModelName-Q4_K_M.gguf \
  "https://huggingface.co/org/ModelName-GGUF/resolve/main/ModelName-Q4_K_M.gguf"
```

The `-C -` flag auto-detects the offset and resumes. Run resumed downloads in parallel too — they'll pick up where they left off.

**Expected sizes for common quants:**
| Model Size | Quant | Approx Size |
|---|---|---|
| 14B dense | Q5_K_M | ~10 GB |
| 14B MoE (3.5B active) | MXFP4 | ~8-9 GB |
| 30B MoE (3B active) | Q4_K_M | ~18-19 GB |
| 35B MoE (3.6B active) | Q4_K_S | ~20 GB |

### Organize downloads

```bash
mkdir -p /models/llm/speedtest
# All downloads go here; cleanup after benchmarking
```

## 2. Verify GGUF Integrity

After downloading, validate each file's GGUF header:

```python
import struct

def check_gguf(path):
    with open(path, 'rb') as f:
        magic = f.read(4)
        if magic != b'GGUF':
            return f'INVALID MAGIC ({magic})'
        ver = struct.unpack('<I', f.read(4))[0]
        tensor_count = struct.unpack('<Q', f.read(8))[0]
        meta_count = struct.unpack('<Q', f.read(8))[0]
        return f'GGUF v{ver}, {tensor_count} tensors, {meta_count} metadata'

# Usage:
# check_gguf('/models/llm/speedtest/model.gguf')
# → 'GGUF v3, 579 tensors, 44 metadata'
```

A valid GGUF file starts with the 4-byte magic `GGUF` (0x47 0x47 0x55 0x46). The header contains version, tensor count, and metadata entry count. If these parse cleanly, the model is not truncated.

This is faster than loading the model — a quick structural check before spending time launching llama-server.

## 3. Speed Benchmark via llama-server Log Parsing

### Why log parsing?

The `/completion` API response may have timing fields under different names depending on build version. The server log always writes human-readable timing lines that are easy to parse:

```
prompt eval time =     933.60 ms /    73 tokens (   12.79 ms per token,    78.19 tokens per second)
       eval time =    2003.41 ms /    50 tokens (   40.07 ms per token,    24.96 tokens per second)
      total time =    2937.01 ms /   123 tokens
```

### Benchmark script pattern

```bash
#!/bin/bash
# Per-model benchmark runner

MODEL="/models/llm/speedtest/MyModel.gguf"
PORT=8091
LLAMA_SERVER="/usr/local/bin/llama-server"
LOGFILE="/tmp/llama-bench.log"
PROMPT='Write a detailed explanation of RISC-V vs ARM architecture...'

# 1. Start server (redirect all output to logfile)
$LLAMA_SERVER \
  -m "$MODEL" \
  --port "$PORT" \
  --host 0.0.0.0 \
  --cache-type-k turbo4 --cache-type-v turbo4 \
  -ngl 99 --no-mmap --mlock --jinja \
  --chat-template-kwargs '{"enable_thinking":false}' \
  --prio 2 --poll 80 --cont-batching --timeout 120 \
  -c 8192 --ctx-size 8192 \
  --threads 16 --threads-batch 32 \
  --parallel 1 --batch-size 2048 --ubatch-size 512 \
  --temp 0.7 --top-p 0.9 --min-p 0.05 --top-k 40 \
  --metrics \
  --n-cpu-moe 128 \
  > "$LOGFILE" 2>&1 &
SERVER_PID=$!

# 2. Wait for health
for i in $(seq 1 180); do
  curl -sf http://localhost:$PORT/health > /dev/null 2>&1 && break
  sleep 1
done

# 3. Warmup (50 tokens)
curl -s -X POST "http://localhost:$PORT/completion" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg prompt "$PROMPT" \
    '{prompt: $prompt, n_predict: 50, temperature: 0.7, cache_prompt: false}')"

# 4. Run 1 (150 tokens)
curl -s -X POST "http://localhost:$PORT/completion" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg prompt "$PROMPT" \
    '{prompt: $prompt, n_predict: 150, temperature: 0.7, cache_prompt: false}')"

# 5. Run 2 (300 tokens)
curl -s -X POST "http://localhost:$PORT/completion" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg prompt "$PROMPT" \
    '{prompt: $prompt, n_predict: 300, temperature: 0.7, cache_prompt: false}')"

# 6. Stop server
sleep 3
kill $SERVER_PID 2>/dev/null
wait $SERVER_PID 2>/dev/null

# 7. Parse log

### Parse results with the reusable script (preferred)

A dedicated parser script lives in the llama-cpp skill:

```bash
cd ~/.hermes/skills/mlops/inference/llama-cpp
python3 scripts/parse-benchmark-log.py /tmp/llama-bench.log "ModelName"
```

It handles line-by-line parsing (avoids the `--` split pitfall below) and properly separates prompt eval from generation eval. No dependencies beyond stdlib.

### Manual parsing (fallback — Python inline)

If the script is unavailable, use this Python code directly:

```python
import re

prompt_results = []
gen_results = []

with open("/tmp/llama-bench.log") as f:
    for line in f:
        pm = re.search(
            r'prompt eval time =\s+\S+\s+ms\s+/\s+(\d+)\s+tokens\s+\(\s+\S+\s+ms per token,\s+([\d.]+)\s+tokens per second\)',
            line
        )
        if pm:
            prompt_results.append((pm.group(1), pm.group(2)))
        
        # Skip lines starting with "prompt" — those were already handled above
        if line.strip().startswith('prompt'):
            continue
        gm = re.search(
            r'eval time =\s+\S+\s+ms\s+/\s+(\d+)\s+tokens\s+\(\s+\S+\s+ms per token,\s+([\d.]+)\s+tokens per second\)',
            line
        )
        if gm:
            gen_results.append((gm.group(1), gm.group(2)))

print("Prompt eval runs:")
for tok, tps in prompt_results:
    print(f"  prompt={tok} tok, {tps} tok/s")
print("Generation runs:")
for tok, tps in gen_results:
    print(f"  gen={tok} tok @ {tps} tok/s")
```

**⚠️ Pitfall — DO NOT split on `--`:** The server log contains `--` in command-line flags (`--cache-type-k`, `--metrics`, `--n-cpu-moe 128`, etc.). `content.split('--')` creates far more blocks than the 3 expected timing blocks, and most will be empty or irrelevant. Always iterate line-by-line.

**Also a pitfall — gen regex matches `prompt eval time` lines:** The regex `r'eval time = ... tokens per second\)'` will match BOTH standalone `eval time =` lines AND `prompt eval time =` lines because "eval time" appears in both. Either skip lines that start with `prompt`, or use a negative lookbehind. The code above handles this with the `line.strip().startswith('prompt')` guard.
```

### MoE vs dense model flags

- **MoE models** (Qwen3-Coder-30B, Qwen3.6-35B-MTP, etc.): Add `--n-cpu-moe 128` (or a value appropriate for the expert count)
- **Dense models** (Qwen2.5-Coder-14B): Omit `--n-cpu-moe` entirely — llama-server rejects this flag with a fatal error on non-MoE models
- **MXFP4 models**: Use the same MoE flags; MXFP4 is a quant format for MoE models
- All models get `-ngl 99` (offload what fits to GPU, rest stays on CPU pinned memory)

### Expected timing breakdown

On a GTX 1080 Ti (11 GB VRAM) with 16-thread Xeon, typical results at 8192 context:

| Model Type | Prompt Eval | Generation (150 tok) | Generation (300 tok) |
|---|---|---|---|
| 30B MoE Q4_K_M | ~50-85 t/s | ~25-28 t/s | ~24-27 t/s |
| 35B MoE Q4_K_S | ~30-85 t/s | ~25-34 t/s | ~25-35 t/s |
| 14B dense Q5_K_M | ~22 t/s¹ | ~8 t/s¹ | ~8 t/s¹ |
| 14B MoE MXFP4 | ~81-90 t/s | ~31 t/s | ~30 t/s |

¹Dense 14B Q5_K_M requires 9.5 GB contiguous CUDA buffer — does not fit in 11 GB VRAM with production services loaded. Must use `-ngl N` (e.g., 25 of 49 layers), which puts ~half the layers on CPU, drastically slowing generation. True full-offload dense 14B speed is higher, but not achievable on this hardware.

Note: Generation speed typically drops 5-10% from 150-token to 300-token runs due to KV cache growth. MTP (Multi-Token Prediction) models may show *higher* generation speed than prompt eval — the MTP architecture predicts multiple tokens per step, boosting throughput even though individual token eval is slower.

## 4. Interpreting Results

### What to look for

- **Prompt eval speed (t/s)** — how fast the model processes your input. Higher is better for chat interactivity. MoE models on this hardware (11GB VRAM, CPU inference for most layers) typically achieve 80-85 t/s on short prompts.
- **Generation speed (t/s)** — how fast the model outputs tokens. This is the critical metric for usability. 20+ t/s feels snappy; 10-15 t/s is acceptable; <10 t/s feels sluggish.
- **Speed stability** — if run #2 (300 tokens) is much slower than run #1 (150 tokens), the model may have a KV cache issue or memory pressure.
- **Model size vs speed** — larger quants (Q5_K_M vs Q4_K_M) and larger models (35B vs 30B) usually trade speed for quality.

### Cross-fleet comparison

Compare new model results against your existing fleet's known benchmarks (see `references/ray-fleet-benchmarks-1080ti.md`):
- Qwen3.6-35B-A3B-UD: ~30 t/s generation (long context benchmark)
- Qwen3.5-9B: ~13 t/s generation
- Gemma-4-26B: ~11 t/s generation

A new model that beats or matches these on your hardware is a viable candidate.

## Pitfalls

- **Server log only, not API response**: The `/completion` JSON response may omit `timings_per_second` or use different field names depending on build version. Always fall back to the server log's `print_timing` lines — they are consistent across builds.
- **`--no-mmap --mlock` loading time**: Large models (15+ GB) can take 30-60 seconds to load due to reading the entire file into pinned memory. Be patient or drop `--mlock` for quick tests.
- **Parallel download truncation**: All curl downloads may stop at the same time when bandwidth adjusts. Resume with `-C -`.
- **Port conflicts**: If you get "couldn't bind HTTP server socket", there's a zombie process. Kill with `lsof -ti :PORT | xargs kill -9`.
- **VRAM verification**: Check free VRAM with `nvidia-smi --query-gpu=memory.free --format=csv,noheader` before loading. An existing production server may consume 5+ GB, leaving <6 GB for testing.
- **MoE flag required**: `--n-cpu-moe` is required for MoE models but rejected by dense models. Classify correctly before starting.
- **Cache reuse**: `cache_prompt: false` in the API request ensures each run starts fresh. Without this, subsequent runs get cached prompt eval results, making the first run look artificially different.