# High-context flag-space notes for Ray's GTX 1080 Ti fleet

Condensed session findings for long-context llama.cpp benchmarking on Ray's Debian + GTX 1080 Ti (11 GB VRAM) box.

## User workflow correction

When evaluating fleet performance, do **not** stop at short-context or ~10k-context prompt tests if the lineup is configured for 64k/100k/128k use.

For this user, benchmark at the **operational context tiers**:
- **65k** for models whose practical floor should be above 60k
- **100k** for models advertised/used as long-context daily drivers
- **128k** when the model/service is configured for it and fit is plausible

Also test **multiple flag variants**, not just one tuned profile, before declaring a winner.

## Real-world research summary

Cross-referenced direction from llama.cpp docs, GitHub-side advice, and community reports:

- `--batch-size` mainly improves **prompt/prefill throughput**, not decode speed.
- `--ubatch-size` is the main **fit/stability** knob when larger batch sizes hit memory pressure.
- Long-context fit is usually dominated by **KV cache strategy**.
- `--flash-attn` is commonly a meaningful long-context win when supported, but on Ray's TurboQuant build it may be auto-enabled by KV cache choice.
- Lower-precision / compressed KV is one of the biggest levers for making 60k-100k+ practical on constrained GPUs.
- `--no-kv-offload` should be treated as a benchmark variable only when there is a reason; it is not a universal default in abstract. Ray's fleet still uses it heavily because those service profiles were already validated on this machine.
- RoPE/YaRN settings should follow model-family metadata rather than arbitrary stretching.

## Qwen long-context note

Ray explicitly wants contexts **above 60k** when the model can handle it.

For Qwen-family dense models, compare the currently used more aggressive style against the more conservative YaRN extension:

### Current service style seen in fleet
```bash
--rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 32768 \
-c 65536 --ctx-size 65536
```

### More conservative comparison case
```bash
--rope-scaling yarn --rope-scale 1.6 --yarn-orig-ctx 40960 \
-c 65536 --ctx-size 65536
```

Reason: the second profile better matches the common interpretation of Qwen3 native ~40k context extended to ~65k, and should be tested rather than assuming the 32k→65k stretch is best.

## Measured results captured across sessions

### Qwen3.6-35B-A3B-UD at ~95k-token prompt / 100k ctx

Same model, same TurboQuant KV, same MoE offload, different batch settings:

#### Variant A
```bash
--batch-size 4096 --ubatch-size 1024
```
- Prompt: **310.10 tok/s** over **95,000** prompt tokens
- Decode: **19.41 tok/s**
- GPU memory line:
  - `11163 = 4703 + (6041 = 4213 + 841 + 986) + 418`

#### Variant B
```bash
--batch-size 2048 --ubatch-size 512
```
- Prompt: **240.54 tok/s** over **95,000** prompt tokens
- Decode: **19.12 tok/s**
- GPU memory line:
  - `11163 = 5205 + (5548 = 4213 + 841 + 493) + 409`

### Practical conclusion

For this machine and model at real long context:
- **4096/1024 is materially better for prefill**
- decode speed is basically unchanged
- this is a concrete confirmation of the general rule: **bigger batch helps prompt ingestion far more than generation**

### Qwen3.5-9B-UD at high context

Two tested high-context variants diverged sharply:

#### Failed variant
```bash
--ctx-size 100000 --batch-size 2048 --ubatch-size 512
```
- prompt run hit **CUDA out of memory** during processing

#### Working fallback
```bash
--ctx-size 100000 --batch-size 1024 --ubatch-size 256
```
- Prompt: **377.52 tok/s** over **60,800** prompt tokens
- Decode: **8.62 tok/s**

### Takeaway

For Ray's 11 GB Pascal box, dense 9B can still be a real long-context model, but **100k context may require shrinking batch/ubatch even when the model itself fits**.

### Qwen3-8B at 65k with two YaRN profiles

Compared:

```bash
--rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 32768
```
vs
```bash
--rope-scaling yarn --rope-scale 1.6 --yarn-orig-ctx 40960
```

Measured results were effectively a wash on this machine:
- `2.0 / 32768`: **177.39 tok/s** prompt, **3.35 tok/s** decode
- `1.6 / 40960`: **177.18 tok/s** prompt, **3.27 tok/s** decode

### Qwen3-14B high-context fit warning

At 65k context, the more aggressive dense-model profile
```bash
--batch-size 2048 --ubatch-size 512
```
failed to load on this 11 GB card. When a 14B long-context run fails to load, the first retry should be to cut batch/ubatch before discarding the model entirely.

## Benchmark design guidance for this user

When updating reports or recommending "best flags," include:
- context size actually tested
- prompt token count actually ingested
- prompt tok/s
- decode tok/s
- memory breakdown if present
- exact variant names, especially:
  - batch/ubatch changes
  - KV cache type changes
  - YaRN/RoPE changes
  - MoE flags like `--n-cpu-moe`

For Ray, a report that only shows one tuned profile is incomplete; include **what different flag sets were tested**.