---
name: system-administration
description: "Comprehensive Linux server administration — system audit & cleanup, security hardening, Docker management, LLM serving optimization, backup strategy, and performance tuning for Debian and Arch/CachyOS."
version: 1.0.0
author: Hermes Agent
license: MIT
platforms: [linux]
metadata:
  hermes:
    tags: [server-admin, system-audit, security, docker, performance, cleanup, linux]
    related_skills: [tailscale-funnel-setup, docker-networking-troubleshooting, hermes-diagnostics]
---

# System Administration

## When to Use
- Disk space low, service crashes, slow performance
- Security hardening (firewall, SSH, Docker)
- Docker container management (limits, logging)
- LLM serving optimization (CUDA, VRAM)
- Backup strategy or "complete audit" requested

## Key Commands

| Area | Command |
|------|---------|
| Disk | `df -h / /home /var /tmp` |
| Docker state | `docker system df`; `docker image prune -f` |
| Resources | `free -h`; `nvidia-smi` |
| Logs | `journalctl --disk-usage`; `journalctl -p err -n 20` |
| Sysctl | `sysctl vm.swappiness` or `cat /proc/sys/vm/swappiness` |
| Ports | `ss -tlnp` |
| Services | `systemctl --failed` |
| Arch cleanup | `sudo paccache -r -k 2`; `sudo pacman -Rns $(pacman -Qdtq)` |

## Cleanup Flow
1. Docker dangling images → unused → build cache
2. Apt autoremove + clean
3. Arch: paccache, AUR cache, orphans, snapper
4. /tmp junk files
5. System tuning (swappiness, journal limits)

**Always verify before removing:** check running containers, active services, and config references. Never remove from `/models/downloads` without checking llama-swap config.

## Pitfalls
- Docker health checks: use `127.0.0.1` not `localhost` (IPv6 resolution)
- Docker mem limits DON'T affect GPU VRAM
- llama-server is NOT in Docker (host process)
- CachyOS swappiness=150 is intentional (ZRAM)
- Never disable SSH password auth without validating key auth first
- RAID0 = zero redundancy
- Separate mounts matter (/data, /models, /backup are different partitions)
- Kill stale PIDs after Docker migration from host networking
- **`no-new-privileges` breaks containers that need sudo:** If `"no-new-privileges": true` is set in `/etc/docker/daemon.json`, any container whose entrypoint runs `sudo` (e.g. strix-sandbox configuring system-wide proxy) exits immediately with code 1. Fix: add `--security-opt no-new-privileges:false` to `create_kwargs` in the container creation code, or remove the daemon-level setting if not needed. The symptom is `docker logs` showing `sudo: The "no new privileges" flag is set, which prevents sudo from running as root.`
- **UFW blocks Docker port bindings:** Changing `127.0.0.1:<port>` to `0.0.0.0:<port>` in docker-compose.yml is NOT enough — UFW's default deny rules still block the port on bridged interfaces. You MUST add a UFW allow rule (`sudo ufw allow from <subnet> to any port <port>`) separately. Verify with `ss -tlnp | grep <port>` showing `0.0.0.0:<port>` AND a successful curl from a different machine/interface.
- **Tailscale Serve TLS on non-443 ports:** Browsers reject Tailscale's internal CA on non-standard ports with "Secure Connection Failed — certificate is not valid". Tailscale only auto-trusts its CA on port 443. For tailnet-only access on non-443 ports, use plain HTTP on the LAN with a UFW allow rule instead.
- **CachyOS fish shell:** SSH commands with `$?`, complex grep patterns, or escaped paths fail. Wrap multi-step commands in `bash -c '...'`. Use Python subprocess or execute_code for complex sequences.

## Exposing Services with Tailscale Serve

When a Docker container binds to `127.0.0.1:<port>` (localhost-only for security), it's unreachable from other devices on the tailnet or LAN. **Tailscale Serve** exposes it to the tailnet without changing docker bindings or opening firewall ports.

### Usage
```bash
# Expose a localhost-bound service on the tailnet (port 443, or specify --https=<port>)
sudo tailscale serve --https=8443 http://127.0.0.1:3081

# Make it permanent across reboots
sudo tailscale serve --bg --https=8443 http://127.0.0.1:3081
```

### Rules
- **Serve** = tailnet-only. **Funnel** = internet-facing. Use Serve for internal services.
- If Funnel already owns port 443, use a different port with `--https=<port>`.
- No docker-compose changes needed — the container stays bound to localhost.
- Access at `https://<hostname>.ts.net:<port>/` from any tailnet device.

### Check existing config
```bash
sudo tailscale serve status
sudo tailscale funnel status
```

## Hermes Token Optimization

Hermes has ~13.9K fixed tokens overhead per API call (73% of each request). Optimize by:

1. **Trim AGENTS.md** — remove stale model lists, old build refs, duplicate info. Target < 2.5K chars.
2. **Consolidate memory** — purge failed experiment notes, condense verbose fleet listings. Target < 1.5K chars (50% capacity).
3. **Restrict `toolsets`** in config.yaml — default loads 31 tools (8,759 tokens). Most sessions need 5-7 (`file`, `terminal`, `skills`, `memory`, `session_search`, `todo`, `clarify`).
4. **Set `platform_toolsets`** per platform — prevents browser tools on messaging (saves 1,258 tokens/req).
5. **Tune compression**: `threshold: 0.3`, `protect_last_n: 10`, `hygiene_hard_message_limit: 400`.
6. See `references/hermes-token-overhead-audit.md` for full methodology.

## Hermes Session DB Cleanup

Prune short-lived sessions (empty stubs, subagent one-shots) from `~/.hermes/state.db`:

```python
# Delete sessions with ≤10 messages
import sqlite3
db = sqlite3.connect("/home/rurouni/.hermes/state.db")
ids = [r[0] for r in db.execute(
    "SELECT id FROM sessions WHERE message_count IS NULL OR message_count <= 10"
).fetchall()]
for sid in ids:
    db.execute("DELETE FROM messages WHERE session_id = ?", (sid,))
placeholders = ','.join('?' for _ in ids)
db.execute(f"DELETE FROM sessions WHERE id IN ({placeholders})", ids)
db.commit()
print(f"Removed {len(ids)} sessions")
```

See `references/hermes-session-db-cleanup.md` for more patterns (age-based, source-based).

## Reference Files
- `references/hermes-token-overhead-audit.md` — Token audit methodology, tool overhead breakdown, optimization checklist
- `references/commands.md` — Full audit commands, cleanup targets table, coding agent removal, compromise audit, boot recovery
- `references/config-templates.md` — sysctl, NVMe scheduler, journald, Docker daemon, PostgreSQL, Redis, Nginx, UFW, SSH hardening templates
- `references/docker-procedures.md` — Resource limits calculation, recommended limits, binding hardening
- `references/server-audit-checklist.md` — Complete audit checklist with commands
- `references/cachyos-emergency-mode.md` — Arch/CachyOS boot recovery (emergency mode, vfat module missing from initramfs, NVIDIA DKMS rebuild)
- `references/cachyos-snapshot-restore.md` — Btrfs snapshot restore when Limine hooks fail; subvolume swap technique; fish shell SSH pitfalls
- `references/cachyos-snapper-limine-sync-failure.md` — `limine-snapper-sync` fails because `/.snapshots` is not a Btrfs subvolume
- `references/nvidia-black-screen-login.md` — KDE black screen after login with NVIDIA beta drivers
- `references/http-ssh-bridge.md` — Serve files via HTTP when SSH PAM/auth hangs
- `references/self-signed-https-local-services.md` — HTTPS for local Docker services
- `references/docker-ufw-port-binding.md` — UFW blocks Docker 0.0.0.0 port bindings; why both steps are needed
- `references/backup-strategy.md` — Automated backup setup
- `references/open-webui-admin-recovery.md` — Reset Open WebUI admin credentials via SQLite DB dump
- `references/open-webui-model-routing.md` — Diagnose model routing mismatches: why response comes from wrong model when dropdown shows the right one; checking Docker env, SQLite DB (config/chat/chat_message tables), llama-swap logs, thinking-toggle interaction
- `references/strix-ai-security-agent.md` — Strix AI penetration testing tool: install (pipx), config (DeepSeek / local llama-swap), Docker sandbox, scan modes, CI/CD integration
- `templates/` — Optimized configs for nginx, Docker, sysctl, PostgreSQL