# Ray local GGUF fleet notes

Session-derived reference for future local model replacements on Ray's Dell Precision 5810 / GTX 1080 Ti stack.

## Current replacement pattern

When replacing local models, update all of these together:

1. Download exact GGUFs with `hf download` (not deprecated `huggingface-cli`).
2. Create/update one systemd service per model under `/etc/systemd/system/`.
3. Update `~/.hermes/scripts/models.py` (single source for `model-switcher` + auto switcher).
4. Update `~/.hermes/config.yaml` `custom_providers[local].models`.
5. Keep SmolLM3 on CPU-only port 8087 for Hindsight unless Ray explicitly says otherwise. Do **not** put this service in `MODELS` / model-switcher, because it should run concurrently with the GPU-swapped models and must not be stopped during swaps. Hindsight points to:
   - model: `HuggingFaceTB_SmolLM3-3B-Q4_K_M.gguf`
   - base URL: `http://127.0.0.1:8087/v1`
   - files to update if needed: `~/.hermes/config.yaml` `hindsight.*`, `~/.hermes/hindsight/config.json`, `~/.hindsight/profiles/hermes.env`
6. Verify each alias with `python3 ~/.hermes/scripts/model-switcher <alias>` and a direct `/v1/chat/completions` probe.
7. Restore the user's active cloud/default model if temporary tests changed `model.default`.
8. Stop local services when done unless the user asked to leave one active.
9. Remove superseded model files and stale services after successful verification.

## Download command pattern

```bash
hf download <repo> <filename.gguf> --local-dir /models/llm/<category>
```

`huggingface-cli download` is deprecated in this environment; use `hf download`.

## Qwen3 65K context flags

For Qwen3 8B/14B models with 32K native context, Qwen recommends YaRN for long context. For Hermes' 65K floor:

```bash
--rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 32768 \
-c 65536 --ctx-size 65536
```

For 128K use `--rope-scale 4`.

## Thinking-mode blank-content fix

Symptom: `/v1/chat/completions` returns empty `message.content`, but response text appears in `reasoning_content`, or `/v1/completions` works.

Fix for normal Hermes/gateway/tool consumers:

```bash
--chat-template-kwargs '{"enable_thinking":false}'
```

This fixed Qwen3.5-9B uncensored Q6_K returning blank chat content.

## Current fleet after May 2026 refresh

### Core fleet (always present)
- `35b` / `8081`: `Qwen3.6-35B-A3B-UD-Q4_K_S.gguf`, 128K
- `9b` / `8082`: `Qwen3.5-9b-uncensored-Q4_K_M.gguf`, 128K
- `gemma4` / `8090`: `Gemma-4-26B` `IQ4_XS`, 131K + mmproj + MoE CPU offload

### Speed-test additions (May 13, 2026 — ports 9021-9025)
- `coder30` / `9021`: `Qwen3-Coder-30B-A3B Q4_K_M` — 18.6 GB, MoE, 65K
- `mtp` / `9022`: `Qwen3.6-35B-A3B-MTP UD-Q4_K_S` — 21.4 GB, MoE MTP variant, 65K (34.6 t/s generation — fastest gen in fleet)
- `coder14` / `9023`: `Qwen2.5-Coder-14B Q4_K_M` — 8.4 GB, dense, 65K (fits ngl=99 at 29.2 t/s)
- `2507` / `9024`: `Qwen3-30B-A3B-2507 Q4_K_M` — 18.6 GB, MoE, 65K
- `opus14` / `9025`: `Qwen3.5-14B-A3B-Opus-Distilled MXFP4` — 8.8 GB, MoE, 65K (81.3 t/s prompt eval)

### Archived / legacy (no longer active in daily rotation)
- `8b` / `8086`: `Qwen3-8B-UD-Q6_K_XL.gguf`, 65K YaRN
- `4b` / `8092`: `Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf`, 128K
- `14b` / `8088`: `Qwen3-14B-UD-Q4_K_XL.gguf`, 65K YaRN
- `gemma3` / `8089`: `gemma-3-12b-it-UD-Q5_K_XL.gguf`, 128K + mmproj
- `gemma3qat` / `8091`: `gemma-3-12b-it-qat-Q4_0.gguf`, 128K + mmproj

### Hindsight CPU-only (always running, outside model-switcher)
- Port `8087`: `HuggingFaceTB_SmolLM3-3B-Q4_K_M.gguf`, 16K. Do **not** put in MODELS / model-switcher.
  - Files: `~/.hermes/config.yaml` `hindsight.*`, `~/.hermes/hindsight/config.json`, `~/.hindsight/profiles/hermes.env`
  - base URL: `http://127.0.0.1:8087/v1`

## Verified quick speeds from direct probes

- Qwen3 4B 2507 Q8: ~40–44 tok/s, ~5.6GB VRAM used
- Qwen3 8B Q6: ~31 tok/s, ~7.0GB VRAM used
- Qwen3 14B Q4_XL: ~24–25 tok/s, ~8.8GB VRAM used

## Model availability note

A requested `Qwen3.5-9B-Uncensored-HauhauCS-Aggressive` `UD-Q5_K_XL` GGUF was not found in public HF search/tree APIs. The available higher-quality replacement over Q4_K_M was `Q6_K` from `Manojb/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf`.