# Local Model Validation — 2026-06-12

Tested 7 models on Debian AI server (RTX 2080 Ti 11GB, CUDA 12.6, llama-swap v224).

## Test Results

| Model | Basic | Stream | Long Input | JSON Mode | Tool Call | Responses | CLI | Class |
|-------|-------|--------|------------|-----------|-----------|-----------|-----|-------|
| qwen36-35b-mtp | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | 44s | coding-edit |
| qwopus-9b | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | 20s | coding-edit |
| qwen35-9b-mtp | ✓ | ✓ | ✗ | ✓ | ✗ | ✓ | 7s | coding-edit |
| gemma-12b | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | 8s | coding-edit |
| gemma-26b | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | 42s | coding-edit |
| nex-n2-mini | ✓ | ✓ | ✗ | ✓ | ✗ | ✓ | 52s | coding-edit |
| bonsai-8b | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | 6s | coding-edit |

## Key Findings

- **No GGUF models support tool/function calling** — inherent to llama.cpp. All classified `coding-edit` at best.
- **Benchmark t/s is misleading** — short-prompt tests produce low throughput because output caps at 256 tokens. Actual llama-server MTP throughput is much higher (qwen36 ~48 t/s, gemma-26b ~53 t/s, qwopus ~46 t/s per llama-server metrics).
- **qwen35-9b-mtp and nex-n2-mini fail long-input** — 250K context causes issues despite config.
- **Best performer:** gemma-12b (4.6 t/s short, 8s cold load, 256K ctx, clean structured output).
- **Fastest load:** bonsai-8b (6s cold) and qwen35-9b-mtp (7s cold) — good for quick queries.

## Cold Load Times (first request launches llama-server)

| Model | Cold Load | VRAM | Reason |
|-------|-----------|------|--------|
| qwen36-35b-mtp | ~44s | ~10.5 GB | 21 GB MoE model |
| qwopus-9b | ~20s | ~6 GB | 7 GB coder model |
| qwen35-9b-mtp | ~7s | ~4.5 GB | 5.5 GB smaller model |
| gemma-12b | ~8s | ~5 GB | 6.3 GB QAT model |
| gemma-26b | ~42s | ~10 GB | 13 GB MoE QAT |
| nex-n2-mini | ~52s | ~9.5 GB | 20 GB MoE model |
| bonsai-8b | ~6s | ~1 GB | 1 GB ternary model |

## Recommended Roles

| Role | Model | Rationale |
|------|-------|-----------|
| Primary coding | qwen36-35b-mtp | Best reasoning, 250K ctx, MTP |
| Fast coder | qwopus-9b | Coder-tuned 9B, Q6_K, faster load |
| Data processing | gemma-12b | 256K ctx, quality output, QAT |
| Quick helper | qwen35-9b-mtp | 7s cold load, uncensored |
| Fast+cheap | bonsai-8b | 1 GB VRAM, 1-bit ternary, fast but limited reasoning |