# June 27, 2026 — Model Onboarding: Ornith-9B, Ornith-35B-Q8 Sweep, Qwen-AgentWorld ## Session 1: Initial Onboarding ### Models Added | Model | File | Size | Type | In llama-swap | |---|---|---|---|---| | Ornith-1.0-9B-Q6_K | `ornith-1.0-9b-Q6_K.gguf` | 6.9 GB | Dense 9B | `ornith-9b-q6` | | Qwen-AgentWorld-35B-UD-Q6_K | `Qwen-AgentWorld-35B-A3B-UD-Q6_K.gguf` | 28 GB | A3B MoE | `qwen-agentworld-35b-q6` | | Ornith-1.0-35B-Q8_0 (re-benchmarked) | `ornith-1.0-35b-Q8_0.gguf` | 35 GB | Qwen3.5 MoE | `ornith-35b-q8` (later removed) | ### Ornith-1.0-35B-Q8 — n-cpu-moe Sweep (RTX 2080 Ti 11GB) Benchmark script: `/tmp/bench-ornith-ncmoe.sh` (bash, standalone, reusable) **Methodology:** - Direct llama-server (not through llama-swap), clean VRAM each run - Values tested: 40 → 38 → 36 → 32 → 28 → 24 - 3 runs per value, 100-token generation, 128K context, Q8 KV cache, `--no-kv-offload` - VRAM recorded via `nvidia-smi` after model load **Results:** | Value | VRAM | TG (avg 3 runs) | OOM? | |---|---|---|---| | 40 (all CPU) | 2.6 GB | 17.00 t/s | No | | 38 | 4.2 GB | 17.48 t/s | No | | 36 | 5.8 GB | 16.19 t/s | No | | **32 ← winner** | **9.1 GB** | **17.95 t/s** | **No** | | 28 | — ❌ | — | OOM (needs 11.8 GB) | | 24 | — ❌ | — | OOM (needs 15.1 GB) | **Q8 OOM floor:** n-cpu-moe < 32 fails on 11GB for Q8_0 (35 GB file). ### Ornith-1.0-9B-Q6 — Initial Config Initially used `--reasoning off` with `-c 131072` and no `--no-kv-offload`. Measured 76 t/s. ### Qwen-AgentWorld-35B — Initial Config Used `--n-cpu-moe 32` (same as Ornith-Q8), no `--reasoning off` (base/world model). --- ## Session 2: Fleet Changes & Debugging ### Removed Models (llama-swap + Open WebUI) - Ornith-1.0-35B-Q8 (`ornith-35b-q8`) — replaced by Q6 MTP variant - Nemotron-Terminal-14B (`nemotron-term-14b`) - GPT-OSS-20B (`gpt-oss-20b`) ### Updated Open WebUI Model List Open WebUI maintains its own model registry in the SQLite `model` table — new llama-swap models do NOT auto-discover. Fixed by inserting model entries and removing stale ones via `docker exec`. ### Qwen-AgentWorld-35B — n-cpu-moe Sweep **Values tested:** 32 → 28 → 24 → 20 → 16 **Script:** `/tmp/bench-agentworld.sh` | Value | VRAM | TG (avg) | OOM? | |---|---|---|---| | 32 | 7.7 GB | 19.9 t/s | No | | **28 ← winner** | **10.3 GB** | **20.5 t/s** | **No** | | 24 | — | — | ❌ | | 20 | — | — | ❌ | | 16 | — | — | ❌ | AgentWorld at UD-Q6_K (28 GB) has slightly better GPU fit than Ornith Q8 (35 GB), allowing n-cpu-moe 28 vs 32. ### CRITICAL: Ornith-9B Debugging — KV Cache OOM Root Cause **Symptom:** User reports Ornith-9B "keeps failing to complete a coding task" in Open WebUI. Chat history shows empty-content assistant responses and error entries. **Investigation path:** 1. Direct API test: ✅ Works fine (1024-token coding task, 63 t/s) 2. Direct streaming test: ✅ 259 events, clean stream 3. llama-swap logs: `recovered from upstream disconnection during streaming` + `no valid JSON data found in stream` + `connect: connection refused` (port dead = server crash) 4. Open WebUI DB: `chat_message` entries with `error: {"content": ""}` and `done=0` 5. VRAM analysis: `-c 131072` without `--no-kv-offload` → 4.6 GB KV cache + 6.9 GB weights = 11.5 GB > 11 GB **Root cause:** KV cache OOM at 128K context. The llama-server process crashes mid-stream during KV cache expansion, before Open WebUI captures the response. The empty `content: ""` error is the time-out/proxy-gateway wrapper, not a reasoning-tag issue. **False lead initially pursued:** `--reasoning off` conflict with Open WebUI's reasoning parser. Proved wrong by: - Streaming with `--reasoning off`: 0 reasoning events, 257 content events, clean - Streaming without `--reasoning off`: 185 reasoning events + 9 content events per "hello" — excessive thinking tokens fill context - Direct API calls produce full responses both ways **Final config (working):** ```yaml ornith-9b-q6: cmd: "... --jinja --reasoning off -ngl 99 --no-mmap --mlock ... -c 32768 --no-kv-offload ..." ``` - **`--reasoning off` kept** — it works correctly on dense models - **`-c 131072 → 32768`** — reduced context frees 3.5 GB VRAM - **`--no-kv-offload` added** — KV cache to system RAM for headroom - Context reduced because the 9B is a fast coder, not a long-context specialist ### Open WebUI Model Registry: Shared Chat Debugging When a user shares an Open WebUI chat link that shows empty/failed responses: ```bash # 1. Find the shared chat by share_id docker exec open-webui python3 -c " import sqlite3, json conn = sqlite3.connect('/app/backend/data/webui.db') # Get the chat cur = conn.execute('SELECT id, title, chat, meta FROM chat WHERE share_id = ?', ('',)) chat = cur.fetchone() chat_id = chat[0] # Get all messages cur = conn.execute('SELECT id, role, content, error, done, model_id FROM chat_message WHERE chat_id = ? ORDER BY id', (chat_id,)) for r in cur.fetchall(): role = r[1] content = (r[2] or '')[:200] error = r[3] done = r[4] if role == 'user': print(f'[USER] {content}') elif error: print(f'[ERR] error={error}') elif content: print(f'[OK] done={done} {content[:150]}') else: print(f'[EMPTY] done={done}') conn.close() " ``` **Key indicators:** - `error={"content": ""}` → response was empty or OOM-crashed - `done=0` → response never completed (stream cut off) - `model_id` should match the intended model — if it's wrong, the chat was routed differently ### llama-swap Log Analysis for Streaming Failures ```bash journalctl -u llama-swap --no-pager -n 200 2>/dev/null | grep -E "disconnect|no valid JSON|connection refused|502" ``` Patterns: - `recovered from upstream disconnection during streaming` → server crashed mid-stream - `no valid JSON data found in stream` → upstream sent empty/corrupt data - `connection refused` → port dead = server process exited - `non-200 response, status=502` → proxy gateway error - Request duration of exactly 5m+ → AIOHTTP_CLIENT_TIMEOUT hit (300s default) ## HF Download Method All models used the `hf` CLI (replaces deprecated `huggingface-cli`): ```bash hf download / --include "" --local-dir ./ ```