# Performance Optimization Guide

Maximize llama.cpp inference speed and efficiency.

## Practical Benchmarking Workflow

When evaluating a new model, **test don't theorize.** Theoretical recommendations about speed, VRAM fit, and context size are frequently wrong — the only reliable data comes from actually loading the model and measuring.

### Step 1: Check available VRAM

Before loading any test model, verify free GPU memory:

```bash
nvidia-smi --query-gpu=memory.used,memory.free,memory.total --format=csv,noheader
# Example: 3847 MiB, 7318 MiB, 11264 MiB
```

Running services consume VRAM. If other llama-server instances are active, you may have far less free than the card's total. Stop them first:

```bash
# Stop all running services
sudo systemctl stop llama-server.service llama-server-8b.service ...
# Or kill specific PIDs
kill -9 $(pgrep -f "llama-server.*8088") 2>/dev/null
# Then verify VRAM is free
nvidia-smi --query-gpu=memory.used --format=csv,noheader
# Target: <1 GB used (just system overhead + small CPU-only processes)
```

### Step 2: Quick load test with llama-server

Start the server in background with an initial flag guess. Use `-ngl 99` first (offload everything), then reduce if OOM:

```bash
/usr/local/bin/llama-server-turbo \
  -m /path/to/model.gguf \
  --port 8090 --host 127.0.0.1 \
  -ngl 99 \
  --no-mmap --jinja --no-kv-offload --cont-batching \
  -c 32768 --ctx-size 32768 \
  -t 16 --threads-batch 32 --batch-size 2048 --ubatch-size 512 \
  --temp 0.2 --metrics 2>&1
```

**Check load output for:**
- `offloaded N/41 layers to GPU` — confirms how many layers fit
- `cudaMalloc failed: out of memory` — reduce `-ngl` by 5 and retry
- `CUDA_Host model buffer size = X MiB` — how much is on CPU pinned memory (fine)

**Pitfall:** `--no-mmap` + `--mlock` together make loading very slow for large models (9GB+ can take 60+ seconds). For quick tests, drop `--mlock`.

**Pitfall:** If port binding fails with "couldn't bind HTTP server socket", there's a zombie process holding it. Use `fuser -k PORT/tcp` then wait, or use a different port.

### Step 3: VRAM verification

Once server prints "model loaded" and the HTTP listener starts, check actual VRAM usage:

```bash
nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader
# Example: 420968, 6844 MiB — model using 6.8 GB VRAM
```

Compare to card total (11 GB for GTX 1080 Ti). If VRAM is tight (<500 MB free), inference may OOM with large context.

### Step 4: Speed benchmark

**Preferred method: /metrics endpoint (cumulative counters, most accurate)**

The `/metrics` endpoint exposes cumulative counters that survive across requests. Take a snapshot before and after, compute delta:

```bash
# Snapshot A
m1=$(curl -s http://127.0.0.1:PORT/metrics | python3 -c "
import sys
m = {}
for l in sys.stdin:
    if l.startswith('llamacpp:prompt_tokens_total') and not 'HELP' in l:
        m['pt'] = l.split()[-1]
    elif l.startswith('llamacpp:prompt_seconds_total') and not 'HELP' in l:
        m['ps'] = l.split()[-1]
    elif l.startswith('llamacpp:tokens_predicted_total') and not 'HELP' in l:
        m['gt'] = l.split()[-1]
    elif l.startswith('llamacpp:tokens_predicted_seconds_total') and not 'HELP' in l:
        m['gs'] = l.split()[-1]
print(json.dumps(m))
")

# Run request
curl -s http://127.0.0.1:PORT/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Write about AI."}],"max_tokens":200,"temperature":0.2}' > /dev/null

# Snapshot B
m2=$(curl -s http://127.0.0.1:PORT/metrics | python3 -c "same as above")

# Compute deltas
```

**Simpler: one-liner with hardcoded warmup/read pattern:**

```bash
# Warmup (2 runs for CUDA kernel compilation + prompt cache)
for i in 1 2; do
  curl -s http://127.0.0.1:PORT/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"messages":[{"role":"user","content":"Write about AI."}],"max_tokens":200,"temperature":0.2}' > /dev/null
done

# Read metrics, run 1 real, read again, compute
curl -s http://127.0.0.1:PORT/metrics | grep -E "prompt_tokens_total|prompt_seconds_total|tokens_predicted_total|tokens_predicted_seconds_total" > /tmp/a.txt

curl -s http://127.0.0.1:PORT/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Write about AI."}],"max_tokens":200,"temperature":0.2}' > /dev/null

curl -s http://127.0.0.1:PORT/metrics | grep -E "prompt_tokens_total|prompt_seconds_total|tokens_predicted_total|tokens_predicted_seconds_total" > /tmp/b.txt

python3 -c "
import json
with open('/tmp/a.txt') as f: a = {l.split()[0]:float(l.split()[-1]) for l in f if 'llamacpp:' in l and not l.startswith('#')}
with open('/tmp/b.txt') as f: b = {l.split()[0]:float(l.split()[-1]) for l in f if 'llamacpp:' in l and not l.startswith('#')}
pt = int(b.get('llamacpp:prompt_tokens_total',0) - a.get('llamacpp:prompt_tokens_total',0))
ps = b.get('llamacpp:prompt_seconds_total',0) - a.get('llamacpp:prompt_seconds_total',0)
gt = int(b.get('llamacpp:tokens_predicted_total',0) - a.get('llamacpp:tokens_predicted_total',0))
gs = b.get('llamacpp:tokens_predicted_seconds_total',0) - a.get('llamacpp:tokens_predicted_seconds_total',0)
print(f'Prompt: {pt} tokens in {ps:.2f}s = {pt/max(ps,0.001):.0f} t/s')
print(f'Gen:    {gt} tokens in {gs:.2f}s = {gt/max(gs,0.001):.1f} t/s')
"
```

**Fallback: /v1/chat/completions timings field** (less accurate, affected by caching):

```bash
curl -s http://127.0.0.1:PORT/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Write 3 sentences about Python."}],"max_tokens":100,"temperature":0.7,"stream":false}' \
  2>/dev/null | python3 -c "
import json,sys
d=json.load(sys.stdin)
u=d['usage']; t=d['timings']
print(f'Prompt: {u[\"prompt_tokens\"]} tok @ {t[\"prompt_per_second\"]:.1f} t/s')
print(f'Gen: {u[\"completion_tokens\"]} tok @ {t[\"predicted_per_second\"]:.1f} t/s')
print(f'Total: {u[\"total_tokens\"]} tok')
"
```

**Quick one-liner (for side-by-side comparisons):**

For rapid A/B testing of two builds or two configs on the same model, use a consistent prompt and extract timings inline:

```bash
# Example: compare old TQ vs new TQ on Phi-4
# Start each build on the same port (stop one before starting the next)

curl -s -m 60 http://localhost:PORT/v1/chat/completions \
  -d '{"messages":[{"role":"user","content":"Write a short paragraph about machine learning in 3 sentences."}],"max_tokens":100}' \
  | python3 -c "import sys,json; d=json.load(sys.stdin); t=d['timings']; \
    print(f'Prompt: {t[\"prompt_per_second\"]:.1f} t/s | Gen: {t[\"predicted_per_second\"]:.1f} t/s | Tokens: {t[\"predicted_n\"]}')"
```

**Side-by-side comparison methodology (sequential):**

To compare two builds (e.g., old TQ binary vs new Stormrage TQ binary):

1. Ensure all VRAM is free — kill all llama-server processes, confirm `nvidia-smi --query-gpu=memory.free` shows full capacity
2. Start build A on a test port (e.g., 8090) — use `-fit off` if Stormrage fork, use consistent flags otherwise
3. Wait for health check to pass: `curl -s http://localhost:8090/health`
4. Run the quick one-liner above to benchmark
5. Kill build A (confirm VRAM released)
6. Start build B on the same port with same flags
7. Run the exact same benchmark one-liner
8. Compare prompt t/s, gen t/s, and VRAM usage

Key: same port, same model file, same prompt, same max_tokens, same temp — only the binary differs.

**Side-by-side comparison methodology (parallel — different ports):**

To compare two builds simultaneously without restarting between tests:

1. Start build A on port 8090, build B on port 8099 (or any two available ports)
2. Ensure both are healthy: `curl -s http://localhost:8090/health && curl -s http://localhost:8099/health`
3. Run the same benchmark prompt against both ports independently
4. Compare results

This is faster for A/B testing but uses more VRAM (both models loaded). Only works if both models fit in VRAM simultaneously (e.g., Phi-4 14B + small model, or two variants of the same model with different binary builds but the same quant file — the model file is shared via mmap, so both servers can load from the same file without doubling VRAM for weights).

**Key metrics:**
- **`predicted_per_second`** — generation speed (target: >10 t/s for usable, >20 t/s for snappy)
- **Prompt processing** should scale with batch size; slower prompt = CPU bottleneck
- Compare to baseline: existing models on same hardware (e.g., Qwen3.6-35B-A3B MoE ~10-15 t/s)

### Step 5: Context size test for Hermes

Hermes Agent requires at least **64K context** or it errors out:

```
ValueError: Model X has a context window of Y tokens, below minimum 64,000 required
```

**Test whether your target context fits in VRAM:**

```bash
/usr/local/bin/llama-server-turbo -m model.gguf ... -c 65536 --ctx-size 65536 ...
```

If the server starts without OOM errors, it fits. If it crashes during `llama_kv_cache` allocation, reduce context or enable aggressive cache compression.

**Context vs VRAM reality check (11 GB GTX 1080 Ti):**

| Model type | ngl | Context | Outcome |
|---|---|---|---|
| Dense 7B (Q4_K, ~5 GB) | 99 | 64K | ✅ Works |
| Dense 14B (Q4_K, ~9 GB) | 99 | 8K | Fits but under Hermes min |
| Dense 14B (Q4_K, ~9 GB) | 25-35 | 32K | Fits, CPU KV offload (15-18 t/s) |
| Dense 24B (Q3_K_M, ~11.5 GB) | any | any | 🔴 Too large for 11 GB |
| MoE 35B (Q4_K_M, 3.6B active) | 99 + n-cpu-moe | 128K | ✅ Works (10-15 t/s) |

For dense models >10 GB on 11 GB card: only viable path is partial offload (`-ngl 20-35`) with CPU KV cache (`--no-kv-offload`). Speed: 10-18 t/s, comparable to MoE models already running.

### Step 6: Cleanup

After exploratory builds, thorough cleanup prevents stale binaries and dangling dependencies:

```bash
# Kill test server
kill -9 $(lsof -ti:PORT)
# Verify VRAM released
nvidia-smi --query-gpu=memory.used --format=csv,noheader
# Should return to baseline (~800-900 MB)
```

**Build trees and orphan binaries:**
- Remove temp clone/build dirs: `rm -rf /tmp/llama.cpp-turboquant /tmp/vanilla-tq` (anything in `/tmp/` created during the session)
- Check for orphan binaries installed during testing: `ls -la /usr/local/bin/llama-server*` — look for test-only binaries that aren't part of the permanent lineup
- For dynamically-linked binaries: verify the build tree they depend on still exists. If the build tree is gone, the binary is dead weight
- Keep only what's worth keeping: the current default plus well-tested alternatives (e.g., old TQ for SWA models, new TQ for non-SWA)

```bash
# Final verification
ls -la /usr/local/bin/llama-server*
nvidia-smi --query-gpu=memory.used --format=csv,noheader
# Restart normal services if needed
```

## CPU Optimization

### Thread tuning
```bash
./llama-cli -m model.gguf -t 8
-t 16  # Physical cores for 16C/32T Xeon
# Avoid hyperthreading (slower for matrix ops)
```

### BLAS acceleration
```bash
make LLAMA_OPENBLAS=1  # 2-3x speedup
```

## GPU Offloading

### Layer offloading
```bash
./llama-cli -m model.gguf -ngl 35  # Hybrid mode
./llama-cli -m model.gguf -ngl 999  # All layers
# Start with -ngl 999, reduce by 5 if OOM
```

### Memory usage
```bash
nvidia-smi dmon  # Watch live VRAM
```

## Batch Processing

```bash
./llama-cli -m model.gguf -b 512   # Batch size
--ubatch 128                        # Physical batch (GPU)
```

## Benchmarks

### Real-World: GTX 1080 Ti (11 GB VRAM, Xeon 16C/32T)

| Model | Config | Gen Speed | Prompt Speed | Context | Notes |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B MoE | `-ngl 99 --n-cpu-moe 48` | 10-15 t/s | ~50 t/s | 128K | Best all-around; MoE efficient |
| **Gemma 4 26B-A4B** | `-ngl 99 --n-cpu-moe 128 --turbo4 --no-think` | **24.3 t/s** | **43 t/s** | **256K** | **Vision on CPU, 8 active experts** |
| Phi-4 14B | `-ngl 99 --no-kv-offload --rope-scale 8` | 25 t/s | 135-208 t/s | 65K | Fastest prompt, best for code |
| Gemma 3 12B | `-ngl 99 --turbo4` | 20.5 t/s | 120-150 t/s | 128K | Most VRAM headroom (3.2 GB free) |
| DeepSeek V4 Flash (cloud) | N/A | 50+ t/s | - | 1M | $0.03/session — cost negligible |

### Turbo4 Cache: Measured Impact

Testing on Gemma 3 12B with 128K context:

| Cache Config | Gen Speed | VRAM | Change |
|-------------|:--------:|:----:|:------:|
| Default (f16) | 19.3 t/s | 8,545 MiB | Baseline |
| `turbo4` (Q4_K + Q8_0) | 24.0 t/s | 7,909 MiB | **+24% gen, -636 MiB VRAM** |

Turbo4 is effectively free performance — faster generation AND lower VRAM. Always enable on 11GB GPUs.

### KV Cache Strategy for MoE Models

MoE models with expert CPU-offloading (`--n-cpu-moe`) already use minimal VRAM for weights (1.5-2.5 GB) but need large compute buffers for the MoE routing logic. This means VRAM savings from turbo4 are less dramatic for MoE than for dense models, but still worthwhile.

| MoE Model | Cache Config | VRAM | Gen Speed |
|-----------|:-----------:|:----:|:--------:|
| Gemma 4 26B-A4B | turbo4 | 10,339 MiB | 24.3 t/s |
| Qwen3.6-35B | turbo4 | ~10 GB | 10-15 t/s |

The compute buffers for 128-expert routing consume ~8.7 GB on Gemma 4, dwarfing the model weights. This is a fixed cost of the architecture, not tunable via flags.