# Flag Design Guide for Local GGUF Models

How to design optimal `llama-server` flags for any GGUF model on a given system.

## The Decision Framework

For every model, determine:

```
Flags = BaseFlags + ArchitectureFlags + ContextFlags + HWFlags
```

### 1. Architecture: MoE or Dense?

| Check | Dense | MoE |
|-------|-------|-----|
| File size relative to params | ~`params * bytes_per_param` | Smaller per parameter (experts are conditional) |
| Samples in `/models/llm/` | Qwen2.5-Coder-7B, Qwen3-8B, Qwen3.5-9B, SmolLM3-3B, Llama-3.2-3B | Qwen3.6-35B-A3B, Qwen3-Coder-30B-A3B |
| HuggingFace `config.json` key | `architectures: ["Qwen2ForCausalLM"]` | `architectures: ["Qwen3MoeForCausalLM"]` + `num_experts` / `num_experts_per_tok` |
| Flag needed | `-ngl 99` (offload all layers to GPU) | `-ngl 99` + `--n-cpu-moe N` (offload expert FFN to CPU, N = number of MoE layers) |

**Rule of thumb:** If the model file is larger than your VRAM (11 GB for GTX 1080 Ti), it's almost certainly MoE or needs offloading.

### 2. MoE: Setting `--n-cpu-moe`

The `--n-cpu-moe N` flag keeps the Mixture-of-Experts FFN weights of the first N layers on CPU. This dramatically reduces VRAM usage because expert weights are the bulk of the model.

- **Set N = num_hidden_layers** (all layers offload MoE to CPU)
- Shared/attention weights still live on GPU via `-ngl 99`
- VRAM usage drops to: shared weights + embedding + KV cache ≈ 3-7 GB depending on model

**Why N = all layers?** Because if the model doesn't fit in VRAM with MoE on GPU, there's no reason to keep ANY expert layers on GPU. The attention and shared layers are the part that benefits from GPU speed, and those still get offloaded.

Examples:
- Qwen3.6-35B (48 layers): `--n-cpu-moe 48`
- Qwen3-Coder-30B-A3B (48 layers): `--n-cpu-moe 48`

### 3. Context Sizing: The VRAM Budget

```
VRAM_budget = 11 GB (GTX 1080 Ti)
KV_cache_per_token = 4 * n_layers * n_kv_heads * head_dim bytes
                  -- With turbo4: ~75% of fp16 size

Context_size = (VRAM_budget - model_on_gpu) / KV_cache_per_token
```

**head_dim** is typically `n_embd / n_heads`. Most models use 128. Qwen MoE models (n_embd=2048, n_heads=32) use 64.

**Turbo4** uses Q4_K for K cache and Q8_0 for V cache — roughly 75% of fp16 KV cache size (12 bytes per value instead of 16).

#### MoE Models (with --n-cpu-moe N = all layers)

| Model | Model on GPU | KV/token (turbo4) | @128K context | Headroom |
|-------|:-----------:|:----------------:|:------------:|:--------:|
| Qwen3.6-35B | ~2.5 GB | 4 * 48 * 4 * 64 * 0.75 = 36 KB | 4.7 GB → ~7.2 GB | ~3.8 GB |
| Qwen3-Coder-30B | ~2.5 GB | 4 * 48 * 4 * 64 * 0.75 = 36 KB | 4.7 GB → ~7.2 GB | ~3.8 GB |

→ **128K is safe** for both MoE models.

#### Dense Models (full GPU offload with -ngl 99)

| Model | File Size | Layers | KV Heads | head_dim | KV/token (turbo4) | Model + @32K | @64K | @128K |
|-------|:--------:|:------:|:--------:|:--------:|:----------------:|:-----------:|:----:|:-----:|
| Qwen2.5-Coder-7B | 5.07 GB | 28 | 4 | 128 | 4 * 28 * 4 * 128 * 0.75 = 43 KB | 5.07 + 1.4 = **6.5 GB** ✅ | 5.07 + 2.8 = 7.9 GB ✅ | 🔴 too much |
| Qwen3-8B | 5.45 GB | 36 | 8 | 128 | 4 * 36 * 8 * 128 * 0.75 = 111 KB | 5.45 + 3.6 = **9.0 GB** ✅ | 🔴 too much | 🔴 |
| Qwen3.5-9B | 5.23 GB | ~48 | 4 | ~85* | ~44 KB | 5.23 + 1.4 = 6.6 GB ✅ | 5.23 + 2.8 = **8.0 GB** ✅ | 🔴 |
| SmolLM3-3B | 2.36 GB | ~30 | ~4 | ~80 | ~29 KB | 2.36 + 1.0 = 3.4 GB ✅ | ✅ | ✅ |
| Llama-3.2-3B | 2.16 GB | 28 | 8 | 128 | 4 * 28 * 8 * 128 * 0.75 = 86 KB | 2.16 + 2.8 = **5.0 GB** ✅ | 2.16 + 5.6 = 7.8 GB ✅ | 2.16 + 11.2 = 13.4 GB 🔴 |

*head_dim estimate = n_embd/n_heads. Qwen3.5-9B uses n_embd ~4096, n_heads ~48 → head_dim ~85.

**Decision rules:**
- ✅ **Green:** Use the model's native max context (or 128K if supported)
- 🟡 **Yellow (comfortable fit):** Inline as safe
- 🔴 **Red:** Reduce context until it fits, or accept CPU paging overhead

### 4. Batch Size — Empirical Testing

On GTX 1080 Ti, batch size beyond 2048/512 doesn't improve speed but uses more VRAM:

| Batch Config | Gemma 3 Gen | Gemma 3 VRAM | When to Use |
|-------------|:----------:|:-----------:|-------------|
| `--batch-size 2048 --ubatch-size 512` | 20.5 t/s | 7,909 MiB | **Default — optimal** |
| `--batch-size 4096 --ubatch-size 1024` | 20.5 t/s | 8,333 MiB (+424 MiB) | Only needed with prompts >4K tokens |
| `--batch-size 1024 --ubatch-size 256` | ~19 t/s | ~7,600 MiB | VRAM emergency mode |

**Recommendation:** Stay at 2048/512. The larger batch doesn't improve throughput on this GPU because PCIe 3.0 bandwidth and CUDA core count (2,560) are the actual bottlenecks.

### 4. Common Base Flags (proven stable)

These come from the user's Qwen3.6-35B service which has been running for weeks without issues:

```ini
--host 0.0.0.0
--cache-type-k turbo4
--cache-type-v turbo4
--no-mmap
--mlock
--jinja
--cont-batching
--timeout 300
--threads 16
--threads-batch 32
--parallel 1
--batch-size 4096
--ubatch-size 1024
--temp 0.2
--top-p 0.95
--min-p 0.05
--top-k 20
--prio 2
--poll 80
--metrics
```

The sampler flags (temp, top-p, min-p, top-k) are user preference, not hardware optimization. Keep them consistent across all models.

### 5. Flash Attention (`--fa`) — GPU Compute Capability Matters

`--fa` (flash attention) requires NVIDIA GPU compute capability 7.0+. The GTX 1080 Ti is CC 6.1 and explicitly rejecting `--fa` returns:
```text
error: invalid argument: --fa
```

**However, some architectures still get FA despite the flag being rejected.** llama-server can auto-detect flash attention support for specific architectures. Check server logs:
```
Flash Attention was auto, set to enabled
```

**Known behavior on GTX 1080 Ti (CC 6.1):**
- **Phi-4 14B (phi3 arch):** No FA, flag rejected, no auto-detect. Gen speed still ~25 t/s at 65K context.
- **Gemma 3 12B (gemma3 arch):** FA flag rejected explicitly, but server **auto-detects and enables it**. This works because Gemma 3's GQA architecture has FA kernels compiled for CC 6.1 while Phi-4's doesn't.
- **Qwen MoE models:** No FA support at all on this hardware.

**Rule for this hardware:** Never pass `--fa` explicitly. Check server logs for "Flash Attention was auto" to know if a specific model gets it anyway. Performance without FA is still excellent at 20-25 t/s for 12-14B models.

### 6. KV Cache Strategy: On-GPU vs CPU Offload

Models running 64K+ context may exceed VRAM for KV cache even with turbo4. Two strategies:

| Strategy | VRAM Usage | Performance | When to Use |
|----------|-----------|-------------|-------------|
| `--no-kv-offload` (default on most setups) | Model weights only + small KV buffer | Attention runs on CPU (slower at very long contexts) | 64K+ on dense models where VRAM is tight |
| KV on GPU (no flag needed) | Model + full KV in VRAM | Fastest attention | Context ≤32K, or small models with room |
| `--no-kv-offload` + turbo4 | Model + compressed KV on CPU | Middle ground | **Preferred for 64K on 11GB VRAM** |

**For Phi-4 14B at 65K on 11GB VRAM:**
- Model at Q4_K: ~8.8GB
- turbo4 KV at 65K if on GPU: needs ~3.4GB → doesn't fit (only ~2.3GB free)
- Solution: `--no-kv-offload` keeps KV on CPU RAM (62GB available)
- Attention still runs on GPU since model weights stay there
- Result: ~25 t/s generation — no measurable slowdown

### 7. Threading Strategy

For Xeon E5-2697A v4 (16C/32T, Broadwell-EP):

| Flag | Value | Rationale |
|------|-------|-----------|
| `--threads` | **16** | Physical cores. Hyperthreads don't help prompt processing. |
| `--threads-batch` | **32** | All 32 logical cores for batch/prompt processing which scales well. |
| `--prio` | **2** | High priority. This is a dedicated AI machine. |
| `--poll` | **80** | Busy-wait polling reduces latency. 80% polling means 80% of wait time is spinning vs yielding. |

## Per-Model Flag Tables

### Gemma 3 12B (Google, dense, 12B)

| Flag | Value | Reason |
|------|-------|--------|
| `-ngl` | `99` | All 42 layers on GPU. Model is only 6.8 GB. |
| `--no-kv-offload` | Required | Model uses ~7.7 GB VRAM, leaving 3.3 GB for other use. KV at 128K needs ~5 GB even with turbo4. |
| `--cache-type-k/v` | `turbo4` | Saves ~640 MB VRAM vs default, 24% faster generation (20.5 t/s vs 19.3 t/s). |
| `--fa` | Omit | Auto-detected and enabled for Gemma 3 on CC 6.1. No explicit flag needed. |
| `-c` / `--ctx-size` | `131072` | Native 128K context. No rope scaling needed. |

**VRAM layout at 128K turbo4:**
- Model file: 6.8 GB
- VRAM used: 7,909 MiB (7.7 GB)
- Free: 3,256 MiB (3.2 GB)
- KV cache on CPU with `--no-kv-offload`

**Performance (GTX 1080 Ti, 260-token prompt):**
- Generation: 20.5 t/s
- Prompt: 120-150 t/s

### Gemma 4 26B-A4B (Google, MoE, 26B total / 4B active per token)

| Flag | Value | Reason |
|------|-------|--------|
| `-ngl` | `99` | All non-expert layers on GPU (31/31 layers offloaded). |
| `--n-cpu-moe` | `128` | Keep all 128 MoE experts on CPU. Expert weights are 14.4 GB and won't fit GPU. |
| `--no-kv-offload` | Required | VRAM is already tight at ~10.3 GB. |
| `--cache-type-k/v` | `turbo4` | Compresses KV cache, critical for 128K+ context on this model. |
| `--chat-template-kwargs` | `'{"enable_thinking":false}'` | Without this, the model outputs reasoning before answering (slower). |
| `--mmproj` | `mmproj-F16.gguf` | Required for vision support. Loads the vision encoder. |
| `--no-mmproj-offload` | Required | Keeps the vision encoder on CPU (1.1 GB). GPU has no room at 10.3 GB used. |
| `-c` / `--ctx-size` | `131072` | Can go up to 262,144 native context. |

**Architecture:**
- 30 transformer blocks, 128 total experts, 8 active per token
- 256K native context length
- Embedding dim: 2816, 16 attention heads, GQA with per-layer KV heads (can vary)
- Vision: `Gemma4ForConditionalGeneration` — supports image input via `--mmproj`

**VRAM layout at 128K turbo4 with vision:**
- Non-expert weights on GPU: 1,573.63 MiB (1.5 GB)
- Expert weights on CPU (CUDA_Host): 14,429.26 MiB (14.1 GB)
- CPU pinned model buffer: 577.50 MiB
- Compute buffers + overhead: ~8.7 GB
- Total VRAM: 10,339 MiB (10.1 GB) — 826 MiB free
- Vision encoder on CPU: 1,137.77 MiB (1.1 GB)

**Performance (GTX 1080 Ti, thinking disabled):**
- Generation: 24.3 t/s
- Prompt: 37-43 t/s
- Vision: Works via CPU-encoded embeddings fed into GPU model

**Key behaviors:**
- The `chat completions` endpoint returns `reasoning_content` alongside `content` when thinking is enabled. The model thinks first, then answers.
- `{"enable_thinking":false}` suppresses the thinking stage for faster responses.
- Response quality without thinking is excellent for chat and instruction following.

### Qwen3.6-27B (Alibaba, dense, 27B with vision)

| Flag | Value | Reason |
|------|-------|--------|
| `-ngl` | Varies | Dense 27B. Q4_K_M (~15-16 GB) won't fit on 11GB. Try Q3_K_S (~10-11 GB) for full GPU, or partial offload (`-ngl 20-30`) for Q4. |
| `--no-kv-offload` | Required at Q3 | Model + KV won't fit GPU even at 256K target. |
| `--cache-type-k/v` | `turbo4` | Saves critical VRAM. |
| `-c` / `--ctx-size` | `131072` | Native 256K, but 128K is practical on this hardware. |
| `--chat-template-kwargs` | `'{"enable_thinking":false}'` | Suppress thinking tokens (same as Qwen3.5/3.6). |
| `--mmproj` | If vision needed | Has `mmproj-F16.gguf` in unsloth repo. |
| `--no-mmproj-offload` | Required | No GPU room for vision encoder. |

**Architecture:** Dense 27B, 64 layers, 5120 hidden dim, 256K native context, Gated DeltaNet + Gated Attention hybrid. NOT MoE — so no `--n-cpu-moe`.

**Available quants (unsloth):** Q3_K_S, Q3_K_M, Q4_K_M, Q4_0, Q5_K_M, Q6_K, Q8_0, plus UD (Unsloth Dynamic): UD-Q2_K_XL through UD-Q8_K_XL. MTP variant also available.

**Best quant for 11GB:** Q3_K_S (~10-11 GB) for full GPU fit, or UD-Q3_K_XL for Unsloth's improved low-bit quantization.

**Performance estimate:** Untested on this hardware. Dense 27B at Q3_K_S should match or exceed Phi-4 14B quality with more parameters but same ~15-25 t/s range.

| Flag | Value | Reason |
|------|-------|--------|
| `-ngl` | `99` | All 41 layers on GPU. Model is 8.5 GB. |
| `--no-kv-offload` | Required | Model uses ~8.8 GB VRAM. KV at 65K with turbo4 would need ~3.4 GB on GPU — not enough room. |
| `--cache-type-k/v` | `turbo4` | Saves VRAM for context window. |
| `--rope-scale` | `8.0` | Phi-4's native context is 16K. Scale to ~65K for Hermes minimum. |
| `-c` / `--ctx-size` | `65536` | 65K with rope scaling. Can't go higher on 11GB VRAM. |
| `--fa` | Cannot use | GTX 1080 Ti CC 6.1 rejects `--fa` for phi3 architecture. No auto-detect either. Performance without FA is still ~25 t/s. |

**Performance:**
- Generation: 25 t/s
- Prompt: 135-208 t/s
- Best for: coding, math, structured tasks

## Vision Setup on Limited VRAM

When loading a multimodal model on a GPU with <2 GB free VRAM:

1. **Download both files**: the main model GGUF and its `mmproj-*.gguf` projector file
2. **Pass `--mmproj path/to/mmproj.gguf`** at server start
3. **Add `--no-mmproj-offload`** to keep the vision encoder on CPU (llama.cpp default is to offload to GPU)
4. **Send base64-encoded images** as `data:image/jpeg;base64,...` URLs. The server cannot download from HTTPS URLs unless compiled with SSL support.
5. **Check model capabilities**: `curl /v1/models` and check the `capabilities` dict. An empty dict `{}` means vision may still work if `--mmproj` is loaded.

**Pitfall — image format:** Not all image formats work. JPEG works reliably. Some PNG files may fail with "Failed to load image or audio file". Use JPEG for most reliable results.

## Thinking / Reasoning Models

Some models (Gemma 4, Qwen3 with thinking, DeepSeek R1 distill) output reasoning in a separate field before the final answer:

- The chat completions API returns `reasoning_content` alongside `content`
- Generation is 2-3x slower when thinking is enabled (model generates dozens of reasoning tokens)
- **To disable thinking:** pass `--chat-template-kwargs '{"enable_thinking":false}'` to the server
- This is a model-level setting — once the server starts with thinking disabled, all responses are direct
- Without disabling thinking, set `max_tokens` high enough to accommodate both reasoning + response (100+ tokens for short answers, depending on the model)
- **Gemma 4 specific:** The `<|channel>` and `<channel|>` tokens are used internally for the thinking/reasoning channel system. `reasoning_content` maps to the thought channel, `content` maps to the response channel.

## MoE: Setting `--n-cpu-moe`

The `--n-cpu-moe N` flag keeps Mixture-of-Experts FFN weights on CPU rather than GPU. This dramatically reduces VRAM usage because expert weights are the bulk of MoE models.

**Important: N is NOT derived from `num_hidden_layers * num_experts_per_tok`.** Each architecture maps N differently:

| Architecture | N = | Why | Example |
|-------------|-----|-----|---------|
| Qwen-style MoE | `num_hidden_layers` | One MoE layer per transformer block. N=layers = all expert layers on CPU. | Qwen3.6-35B: 48 layers → `--n-cpu-moe 48` |
| Gemma 4-style MoE | `total_expert_count` | Experts are organized as a global pool, not per-layer. N must be the total count from GGUF metadata. | Gemma 4 26B: 128 experts → `--n-cpu-moe 128` |

**How to find the right value:**
1. Check the GGUF metadata via `llama-server` load output — look for `expert_count` or similar
2. For Qwen MoE: `num_hidden_layers` (from `config.json`)
3. For Gemma 4: The total expert count (128) — NOT `30 layers * 8 active = 240`

**Why N matters:** Setting N too low leaves some expert weights on GPU, which may cause OOM for large MoE models. Setting N too high is harmless (excess expert slots are ignored). When in doubt, round UP.

**Architecture check (identifying MoE without looking at config):**
- Dense: File size ≈ `params * bytes_per_param` (e.g., 12B @ Q4_K_M ≈ 6-7 GB)
- MoE: File size is closer to `total_params * bytes_per_param` but with expert offload the GPU usage is far smaller (e.g., 26B @ Q4_K_M ≈ **1.57 GB on GPU** with `--n-cpu-moe 128`)

### 7. Quick Reference: HuggingFace API for Model Specs

Fetch model architecture details without downloading:

```bash
# Get full config:
curl -sL "https://huggingface.co/<org>/<model>/raw/main/config.json"

# Key fields to check:
# - architectures: ["Qwen3MoeForCausalLM"] → MoE, ["Qwen2ForCausalLM"] → Dense
# - num_hidden_layers: count (for --n-cpu-moe and KV calc)
# - num_attention_heads: KV calc (head_dim = n_embd / this)
# - num_key_value_heads: KV calc (for GQA)
# - hidden_size: n_embd
# - max_position_embeddings: native context (may be larger than what you want to run)
# - num_experts / num_experts_per_tok: MoE confirmation

# Via HF API (no raw access needed):
curl -s "https://huggingface.co/api/models/<org>/<model>" | python3 -c "import json,sys; c=json.load(sys.stdin)['config']; print(c.get('num_hidden_layers','?'), c.get('num_attention_heads','?'), c.get('num_key_value_heads','?'), c.get('hidden_size','?'), c.get('max_position_embeddings','?'))"
```