# Model Research & Gap Analysis

Systematic workflow for evaluating what GGUF models to add to Ray's local stack.

## When to Use

- Exploring new models to fill gaps in the current fleet
- User asks for recommendations or "what am I missing out on"
- Evaluating a specific candidate model for fit

## Workflow

### 1. Check What Fits the VRAM Budget

Hardware: GTX 1080 Ti 11GB VRAM, PCIe 3.0 x16. Only one model runs at a time.

**Quick VRAM calculator at Q4_K_M:**
- Dense model: `param_count * 0.5 GB + ~1-1.5 GB overhead`
- MoE model (active params): `active_count * 0.55 GB + ~1 GB overhead` (experts can swap)
- Subtract KV cache: with `--no-kv-offload`, KV is on CPU RAM (62GB) — only model weighs against VRAM

**Fit categories:**
- **Comfy (≤9GB):** Fits with headroom — Gemma 3 12B (~6.8GB), Phi-4 14B (~8.6GB), 7-8B at Q5 (~5.5GB)
- **Tight (9-10.5GB):** Fits but no headroom — Llama-4-Scout 17B (barely fits at Q3)
- **Bust (>10.5GB):** Must CPU offload — Mistral Small 24B (14.5GB at Q4, 11GB at Q3_K_M)

### 2. Search Hugging Face for Candidates

Use the tree API to find actual GGUF files and sizes:

```bash
# Search by model family
curl -s "https://huggingface.co/api/models?search=<model-name>+GGUF&sort=downloads&direction=-1&limit=20" | python3 -c "
import json,sys
data = json.load(sys.stdin)
for m in data[:10]:
    print(f'{m[\"modelId\"]:50s} {m.get(\"downloads\",0):>10,} downloads  {m.get(\"likes\",0):>4} likes')
" 2>/dev/null

# Check a specific quant repo for file sizes
curl -s "https://huggingface.co/api/models/<producer>/<model>-GGUF/tree/main?recursive=true" | python3 -c "
import json,sys
data = json.load(sys.stdin)
for f in data:
    if f['type'] == 'file' and f['path'].endswith('.gguf'):
        gb = f.get('size', 0) / (1024**3)
        print(f'{f[\"path\"]:60s} {gb:6.2f} GB')
"

# Or get actual size via HEAD to LFS URL
# from huggingface_hub import list_repo_files, hf_hub_url
# import requests
# files = list_repo_files('bartowski/gemma-3-12b-it-GGUF')
# for f in files:
#     if 'Q4_K_M' in f:
#         url = hf_hub_url('bartowski/gemma-3-12b-it-GGUF', f)
#         resp = requests.head(url, allow_redirects=True)
#         gb = int(resp.headers['content-length']) / (1024**3)
#         print(f'{f}: {gb:.2f} GB')
```

**Popular GGUF producers** (sort their repos by downloads):
- bartowski (largest, most popular quants)
- MaziyarPanahi (varied and active)
- unsloth (optimized for training)
- lmstudio-community (LM Studio curated)
- tvall43 (Ray's go-to for MoE variants — same maker as Qwen3.6-35B-UD)
- mcxiaoke, QuantFactory, mradermacher

### 3. Check r/LocalLLaMA Community Sentiment

Don't trust model cards. The real signal is in Reddit threads.

**Key search queries:**
- `site:reddit.com/r/LocalLLaMA <model-name> review`
- `site:reddit.com/r/LocalLLaMA <model-name> vs <competitor>`
- `site:reddit.com/r/LocalLLaMA best model 11GB VRAM 2026`
- `site:reddit.com/r/LocalLLaMA <model-name> experience`

**What to extract:**
- Consensus sentiment (positive/mixed/negative)
- Specific weaknesses reported (repetition bugs, tool-calling issues, refusal patterns)
- Real-world speed reports on similar hardware
- Comparison vs known models (Phi-4, Qwen3.5, Llama 3.1)

### 4. Verify with the Fleet Gap Map

Compare candidates against Ray's current coverage:

| Capability | Covered By | Gap? |
|-----------|-----------|------|
| Big brain reasoning | Qwen3.6-35B-A3B (128K) | ✅ Covered |
| Fast everyday | Phi-4 14B (25 t/s), Gemma 3 12B (20 t/s) | ✅ Covered |
| Cloud speed | DeepSeek Flash ($0.03/session) | ✅ Covered |
| Uncensored chat | Qwen3.5-9B | ✅ Covered |
| Tiny/fast | Llama-3.2-3B, SmolLM3-3B (CPU memory) | ✅ Covered |
| Vision (images) | **None local** | ⚠️ Still a gap — Gemma 3 GGUF is text-only, no vision encoder |
| Multilingual | **None local** | 🟢 Mistral NeMo 12B fills this if needed |
| 1M+ context | **None local** | 🟢 Llama-4-Scout 17B but model quality is poor at Q3 |

### 5. Benchmark Before Committing

Always live-test a candidate before creating a systemd service. Use the /metrics endpoint for accurate timing (more reliable than the `timings` field in /chat/completions, which can be affected by prompt caching):

```bash
# 1. Start temp server
sudo pkill -f "llama-server"
/usr/local/bin/llama-server-turbo -m /path/to/model.gguf --port 8090 --host 0.0.0.0 \
  -ngl 99 --no-kv-offload --cache-type-k turbo4 --cache-type-v turbo4 \
  -c 65536 --ctx-size 65536 --batch-size 2048 --ubatch-size 512 \
  --temp 0.2 --jinja --metrics 2>&1 &

# 2. Wait for load (~30-60s)
sleep 90; curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8090/v1/models

# 3. Check VRAM
nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader

# 4. Benchmark using /metrics deltas
curl -s http://127.0.0.1:8090/metrics | grep -E "prompt_tokens_total|prompt_seconds_total|tokens_predicted_total|tokens_predicted_seconds_total"

# 5. Run a generation
curl -s http://127.0.0.1:8090/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Write about AI."}],"max_tokens":200,"temperature":0.2}'

# 6. Read metrics again and calculate deltas
curl -s http://127.0.0.1:8090/metrics | python3 -c "
import sys
m = {}
for l in sys.stdin:
    if 'prompt_tokens_total' in l and not l.startswith('#'):
        m['pt'] = float(l.split()[-1])
    elif 'prompt_seconds_total' in l and not l.startswith('#'):
        m['ps'] = float(l.split()[-1])
    elif 'tokens_predicted_total' in l and not l.startswith('#'):
        m['gt'] = float(l.split()[-1])
    elif 'tokens_predicted_seconds_total' in l and not l.startswith('#'):
        m['gs'] = float(l.split()[-1])
if all(k in m for k in ['pt','ps','gt','gs']):
    print(f'Prompt: {int(m[\"pt\"])}/{m[\"ps\"]:.2f}s = {int(m[\"pt\"])/m[\"ps\"]:.0f} t/s')
    print(f'Gen:    {int(m[\"gt\"])}/{m[\"gs\"]:.2f}s = {int(m[\"gt\"])/m[\"gs\"]:.1f} t/s')
"
```

**Always run 2 warmup requests + 3 measured requests** to get consistent numbers. The first request includes CUDA kernel compilation overhead and prompt cache initialization.

```bash
# Temp test on free port
/usr/local/bin/llama-server-turbo -m /path/to/model.gguf --port 8090 \
  -ngl 99 --no-kv-offload -c 65536 --ctx-size 65536 \
  --cache-type-k turbo4 --cache-type-v turbo4 \
  --temp 0.2 --metrics &
sleep 90  # Wait for load (PCIe 3.0 is slow) 

# Generation speed
curl -s http://127.0.0.1:8090/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Write a paragraph about AI."}],"max_tokens":200,"temperature":0.7}' \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(f'{d[\"usage\"][\"completion_tokens\"]} tok @ {d[\"timings\"][\"predicted_per_second\"]:.1f} t/s')"
```

**Speed bars to clear:** Qwen3.6-35B = 10-12 t/s | Phi-4 = 25 t/s | Gemma 3 12B = 20 t/s | Anything below 8 t/s is too slow for chat use.

**Batch-size tuning finding:** On GTX 1080 Ti, `--batch-size 2048 --ubatch-size 512` is the sweet spot. Increasing to 4096/1024 uses more VRAM (~400 MiB extra) with no measurable speed improvement. Don't go higher.

**Config variant comparison (empirical data from GTX 1080 Ti):**

| Variant | Gemma 3 Gen | Gemma 3 VRAM | Phi-4 Gen | Phi-4 VRAM |
|---------|:----------:|:-----------:|:--------:|:----------:|
| Default f16 cache | 19.3 t/s | 8,545 MiB | ~18 t/s | ~9,000 MiB |
| turbo4 cache | **24.0 t/s** (+24%) | **7,909 MiB** (-640 MiB) | **25 t/s** | **8,821 MiB** |
| turbo4 + 64K ctx | 23.8 t/s | 7,533 MiB | — | — |
| turbo4 + big batch | 20.5 t/s | 8,333 MiB | — | — |

**turbo4 always wins** — faster gen, less VRAM, no downside. Always use `--cache-type-k turbo4 --cache-type-v turbo4` when available.

**Batch-size finding:** `--batch-size 2048 --ubatch-size 512` is the sweet spot on GTX 1080 Ti. 4096/1024 adds ~400 MiB VRAM with zero speed improvement. Don't go higher.

**Context size finding:** 128K context does not penalize generation speed vs 64K — actually equal or slightly faster (cache warmup effects). Always use native context if the model supports it.

**Flash attention on CC 6.1:** The `--fa` flag is rejected on GTX 1080 Ti. However, some model architectures auto-detect and enable FA even without the flag. Gemma 3 12B logs "Flash Attention was auto, set to enabled" while Phi-4 14B does not. Check `journalctl` or server startup logs, not flag success, to determine if FA is active for a given model.

### 6. Note: Model-Switching Flow

The model-switcher (`python3 ~/.hermes/scripts/model-switcher <alias>`) handles:
1. Stop current service
2. Start target service
3. Update Hermes config (provider, model, base_url)
4. Wait for health check on target port
5. User runs `/reset` in Hermes for new session

Always test that `model-switcher <alias>` works end-to-end after adding a new model.

## Hardware Reference

- GPU: GTX 1080 Ti, 11GB VRAM, CC 6.1 (NO flash attention support)
- CPU: Xeon E5-2697A v4, 16C/32T
- RAM: 62GB DDR4
- PCIe: 3.0 x16 (~1.5 GB/s transfer) — model loading over this bus is the bottleneck, not model size
- Storage: Models on NVMe at /models/llm/