# Benchmark Methodology for llama-server

## Hardware
- GPU: NVIDIA RTX 2080 Ti 11GB (Turing, sm_75)
- CPU: Xeon E5-2697A v4, 62GB RAM
- Driver: 550.163.01, CUDA 12.6
- Binary: llama-server-sm75 (EsmaeelNabil fork, feat/mtp-turboquant-kv-cache)

## Test Prompt (consistent across all benchmarks)
```json
{"model":"local","messages":[{"role":"user","content":"Write a Python quicksort function with detailed comments. Return ONLY the code."}],"max_tokens":256,"temperature":0.2,"stream":false}
```

## Flag Variation Matrix (per model)

| Profile | cache K/V | flash-attn | MTP | draft cache | --no-host | -fitt | notes |
|---------|-----------|------------|-----|-------------|-----------|-------|-------|
| A_baseline | f16 | off | off | — | off | 1024 | vanilla baseline |
| B_turbo | turbo3/turbo4 | on | off | — | on | 512 | optimized no-MTP |
| C_MTP | turbo3/turbo4 | on | on (n-max 4) | q4_0/q4_0 | on | 512 | full optimization |
| D_maxctx | turbo3/turbo4 | on | on (n-max 3) | q4_0/q4_0 | on | 512 | push context to limit |

## Context Size Test Targets
- 128K: baseline for all models
- 150K: mid-range push
- 200K: max target (user requirement: must be >150K for Hermes)

## Metrics Collected
- gen_tok/s: generation tokens per second
- prompt_eval tok/s: prompt processing speed
- VRAM before/after: nvidia-smi memory.used
- Startup time: seconds to /health endpoint responding
- Server errors: grep -ci 'error|warn' on server log
- Response validity: character count of content field (non-zero = valid)

## Common Pitfalls During Benchmarking

1. **"&" backgrounding banned in foreground terminal mode**: Always use `terminal(background=true)` for server start, then separate `terminal()` calls for health checks and curl tests.

2. **`--chat-template-kwargs` deprecated**: Use `--reasoning off` instead. The old flag causes empty `content` fields (all output goes to `reasoning_content`).

3. **Second-resolution timing with `date +%s`**: For gen speeds >30 tok/s, second-resolution can give 0-second durations. Use `date +%s%N` for nanosecond precision, divide by 1e9.

4. **Cached model loads**: After killing and restarting the same model, the kernel page cache speeds up the second load. First-cold-start times are the real numbers.

5. **Health check race**: `curl /health` may succeed before the model is fully initialized. Add a 3-second sleep after health check before running the benchmark prompt.

## Benchmark Results (RTX 2080 Ti 11GB)

All tests: turbo3 K, turbo4 V, flash-attn on, --no-host, -fitt 512, --reasoning off.

| Model | Profile | tok/s | ctx | VRAM | startup | MTP | Notes |
|-------|---------|-------|-----|------|---------|-----|-------|
| Heretic 35B Q4_K_M | turbo+FA | 43.5 | 128K | 3.8GB | 42s | ✓ | Fastest overall |
| Qwen3.6 35B Q4_K_M | turbo+FA+MTP | 42.7 | 128K | 4.4GB | 42s | ✓ | Daily driver |
| DeepSeek Flash 9B Q6_K | turbo+FA+MTP | 41.5 | 128K | 8.2GB | 5s | ✓ | Fastest small |
| Qwopus Coder 9B Q6_K | turbo+FA+MTP | 38.9 | 128K | ~7GB | 12s | ✓ | Coding specialist |
| Qwen3.6 35B Q4_K_M | turbo+FA (no MTP) | 36.6 | 128K | 3.3GB | 42s | ✗ | Baseline MoE |
| Qwen3.5 9B Q5_K_XL | turbo+FA | 27.5 | 128K | 6.4GB | 5s | ✗ | Uncensored, slower |
| Gemma 26B Q4_K_XL | turbo+FA | 25.7 | 131K | 3.2GB | 37s | — | MoE, no MTP support |