# LLM Serving Optimization — RTX 2080 Ti

## Hardware Profile

- **GPU:** NVIDIA RTX 2080 Ti (TU102), 11 GB VRAM, SM 7.5
- **CPU:** Intel Xeon E5-2697A v4, 32 threads, 2.6 GHz base
- **RAM:** 64 GB DDR4
- **CUDA:** 12.6, driver 550.163.01

## VRAM Budget

| Model | Quant | Weights | KV Cache (128K) | KV Cache (256K) | Total VRAM |
|-------|-------|---------|-----------------|-----------------|------------|
| Qwen3.6 35B MoE | Q4_K_M | ~22 GB | ~3 GB | ~5 GB | 25-27 GB |
| Gemma 4 26B A4B | Q4_K_M | ~16 GB | ~2 GB | ~3 GB | 18-19 GB |
| Qwopus 9B | Q6_K | ~7 GB | ~1 GB | ~2 GB | 8-9 GB |
| Qwen3.5 9B | Q4_K_M | ~5.5 GB | ~1 GB | ~2 GB | 6.5-7.5 GB |
| Gemma 4 12B | Q4_K_M | ~6.7 GB | ~1 GB | ~2 GB | 7.7-8.7 GB |

**Note:** llama-server loads model weights into host RAM first, then offloads to VRAM. The 28 GB RSS includes both host RAM (weights + KV cache) and GPU memory.

## Optimized llama-server Flags

### For 35B MoE Models (Qwen3.6, Gemma 4 26B)

```bash
-ngl 99                          # Max GPU layers
--no-mmap --mlock                # Pin to RAM
--cache-type-k q8_0 --cache-type-v q8_0  # High-precision KV cache
--flash-attn on                  # Flash attention
--no-cont-batching               # Better for single concurrency
--n-cpu-moe 128                  # MoE parallelism (16 for 26B, 128 for 35B)
--spec-type mtp --spec-draft-n-max 4 --spec-draft-p-min 0.9
--spec-draft-type-k q8_0 --spec-draft-type-v q8_0
--prio 2 --poll 80               # High priority
--timeout 300                    # 5 min idle timeout
--threads 16 --threads-batch 32  # CPU threads
--batch-size 2048 --ubatch-size 512
--temp 0.2 --top-p 0.95 --min-p 0.05 --top-k 20
```

### For 9B Models (Qwopus, Qwen3.5)

```bash
-ngl 99
--no-mmap --mlock
--cache-type-k q8_0 --cache-type-v q8_0
--flash-attn on
--cont-batching                  # OK for smaller models
--no-kv-offload
--spec-type mtp --spec-draft-n-max 4 --spec-draft-p-min 0.9
--spec-draft-type-k q8_0 --spec-draft-type-v q8_0
--prio 2 --poll 80
--timeout 300
--threads 16 --threads-batch 32
--batch-size 1024 --ubatch-size 256
--temp 0.2 --top-p 0.95 --min-p 0.05 --top-k 20
```

### For 12B Dense (Gemma 4 12B)

```bash
-ngl 99
--no-mmap --mlock
--cache-type-k q8_0 --cache-type-v q8_0
--flash-attn on
--cont-batching
--prio 2 --poll 80
--timeout 300
--threads 16 --threads-batch 32
--batch-size 1024 --ubatch-size 256
--temp 0.2 --top-p 0.95 --min-p 0.05 --top-k 64
```

## Context Size vs VRAM

Larger context = more KV cache = more VRAM. Practical limits on 11 GB VRAM:

| Model | Max Practical Context | Reason |
|-------|----------------------|--------|
| 35B MoE | 128K | 256K KV cache exceeds VRAM |
| 26B MoE | 128K | 256K KV cache exceeds VRAM |
| 12B Dense | 256K | Fits within VRAM |
| 9B Dense | 256K | Fits within VRAM |

## Performance Benchmarks (RTX 2080 Ti)

| Model | Tokens/sec | VRAM Used | Host RAM |
|-------|-----------|-----------|----------|
| Qwen3.6 35B MoE | ~48 t/s @ 200K | ~7.6 GB | ~28 GB |
| Qwopus 9B | ~45 t/s @ 150K | ~5 GB | ~8 GB |
| Qwen3.5 9B | ~45 t/s @ 200K | ~5 GB | ~8 GB |
| Gemma 4 12B | ~40 t/s @ 256K | ~6 GB | ~10 GB |
| Gemma 4 26B | ~35 t/s @ 128K | ~7 GB | ~20 GB |

## Model Selection Guide

### Primary Model (35B MoE)
- Best for: Complex reasoning, long context, code generation
- Use when: Quality matters more than speed
- llama-swap TTL: 3600s (keep loaded longer)

### Coder Model (9B MoE)
- Best for: Code generation, fast iteration
- Use when: Speed matters, code tasks
- llama-swap TTL: 1800s

### Uncensored Model (9B Dense)
- Best for: Creative writing, unrestricted generation
- Use when: No content filters needed
- llama-swap TTL: 1800s

### Gemma 4 12B (Dense)
- Best for: General tasks, balanced quality/speed
- Use when: No MTP needed, general purpose
- llama-swap TTL: 3600s

### Gemma 4 26B (MoE)
- Best for: Complex tasks, high quality
- Use when: Need more capability than 9B
- llama-swap TTL: 1800s
- Note: Uses n-cpu-moe=16 (not 128) due to smaller MoE structure

## Pitfalls

- **Don't set --n-cpu-moe too high for small MoE models** — Gemma 4 26B uses n-cpu-moe=16, not 128. The 35B MoE uses 128 because it has more experts.
- **KV cache precision matters** — q8_0 KV cache uses more VRAM but produces better quality than q4_0. For 11 GB VRAM, q8_0 is the sweet spot.
- **--no-cont-batching for primary model** — When serving a single high-quality model, disable continuous batching for better latency. Enable it for smaller models that may serve multiple users.
- **MTP speculative decoding** — --spec-type mtp with --spec-draft-n-max 4 gives ~2x speedup for MoE models. Requires the model to have MTP heads.
- **Context size vs KV cache** — Setting --ctx-size higher than your VRAM can hold for KV cache will cause OOM. Match context size to available VRAM.
- **llama-swap TTL** — Set TTL based on model size. Larger models (35B) should have longer TTL (3600s) to avoid reload overhead. Smaller models (9B) can have shorter TTL (1800s).
- **GPU temperature** — Monitor with `nvidia-smi --query-gpu=temperature.gpu`. Sustained temps above 85°C may indicate cooling issues.
- **Power draw** — RTX 2080 Ti draws ~63W during inference at 250W limit. This is normal. If it draws >200W, something is wrong.