# System Administration — Command Reference

## Phase 1: Audit Commands

```bash
# Disk usage — root mount, key directories, top-level dirs
df -h / /home /var /tmp 2>/dev/null
du -sh /var/log /tmp /var/cache/apt /root /home/<user>/.cache /home/<user>/.local/share/Trash 2>/dev/null
du -sh /* 2>/dev/null | sort -rh | head -20

# User home breakdown
du -sh /home/<user>/* 2>/dev/null | sort -rh | head -20

# Docker state
docker system df
docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}"

# Packages — old kernels, orphans
dpkg -l 'linux-image-*' 2>/dev/null | grep "^ii"
apt-get --dry-run autoremove 2>/dev/null | grep "^Remv"

# System resources
free -h
nvidia-smi --query-gpu=memory.used,temperature.gpu,power.draw --format=csv 2>/dev/null

# Logs
journalctl --disk-usage 2>/dev/null
find /var/log -type f -size +10M -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh | head -10

# Swap and sysctl
sysctl vm.swappiness 2>/dev/null || cat /proc/sys/vm/swappiness
cat /proc/swaps 2>/dev/null

# Snap/Flatpak
snap list 2>/dev/null; flatpak list 2>/dev/null
```

### Arch / CachyOS Package Audit
```bash
pacman -Q | wc -l                        # total packages
pacman -Qe | wc -l                       # explicitly installed
pacman -Qm | wc -l                       # foreign/AUR packages
du -sh /var/cache/pacman/pkg/             # pacman cache
du -sh ~/.cache/yay/ ~/.cache/paru/ 2>/dev/null  # AUR build cache
pacman -Qdt 2>/dev/null                  # orphan packages
uname -r                                  # current kernel version
sudo snapper -c root list 2>/dev/null    # btrfs snapshots (CachyOS)
```

## Phase 2: Deep Audit Commands

### Developer Tool / Coding Agent Inventory
```bash
# PATH binaries
for cmd in cursor codeium windsurf claude copilot aider continue tabby cody codex open-interpreter github-copilot-cli command-code commandcode cmd; do
  path=$(which "$cmd" 2>/dev/null) && echo "$cmd: $path" || echo "$cmd: not found"
done

# Editors/IDEs in PATH
which vim nvim code codium zed emacs helix subl nano 2>/dev/null

# Custom PATH dirs
IFS=: read -ra DIRS <<< "$PATH"
for dir in "${DIRS[@]}"; do
  if [ -d "$dir" ] && [[ "$dir" != /usr/bin && "$dir" != /bin && "$dir" != /usr/sbin && "$dir" != /sbin ]]; then
    ls -1 "$dir" 2>/dev/null
  fi
done

# npm global packages
npm list -g --depth=0 2>/dev/null

# pipx installed tools
pipx list 2>/dev/null

# pip3 for coding tools
pip3 list 2>/dev/null | grep -iE "(cursor|codex|continue|aider|tabby|cody|copilot)"

# Cargo installed tools
ls ~/.cargo/bin/ 2>/dev/null | grep -vE "^(cargo|rustc|rustdoc)"

# Flatpak apps
flatpak list --app 2>/dev/null

# Snap apps
snap list 2>/dev/null

# AppImages in home
find ~/ -maxdepth 3 \( -name "*.AppImage" -o -name "*.appimage" \) 2>/dev/null

# VS Code extensions
code --list-extensions 2>/dev/null

# Desktop entries
ls /usr/share/applications/*cursor* /usr/share/applications/*codex* /usr/share/applications/*claude* /usr/share/applications/*continue* /usr/share/applications/*windsurf* 2>/dev/null

# Config dirs for coding agents
ls -d ~/.cursor* ~/.codex* ~/.continue* ~/.claude* ~/.codeium* ~/.windsurf* ~/.aider* ~/.copilot* 2>/dev/null

# Shell aliases/functions
grep -iE "(cursor|codex|continue|claude|copilot|aider|command.code)" ~/.bashrc ~/.zshrc ~/.config/fish/config.fish 2>/dev/null
```

### Security & Services Audit
```bash
# Network — all listening ports, firewall status
ss -tlnp 2>/dev/null
ufw status verbose 2>/dev/null || echo "ufw not installed"
iptables -L -n -v 2>/dev/null | head -50

# Tailscale
tailscale status 2>/dev/null
tailscale funnels 2>/dev/null

# RAID and storage health
cat /proc/mdstat 2>/dev/null
smartctl -a /dev/nvme0n1 2>/dev/null | grep -E "Model|Temperature|Power_On|Wear_Level|Health"

# GPU / CUDA / LLM stack
nvidia-smi --query-gpu=driver_version,name,memory.total,memory.used --format=csv 2>/dev/null
nvcc --version 2>/dev/null || echo "nvcc not found"
ps aux | grep llama-server | grep -v grep

# SSH
cat /etc/ssh/sshd_config 2>/dev/null | grep -v "^#" | grep -v "^$"
fail2ban-client status 2>/dev/null

# Backup
find / -name "*backup*" -type f 2>/dev/null | grep -v proc | grep -v sys | head -10
crontab -l 2>/dev/null

# Crash history
last reboot 2>/dev/null | head -5
journalctl --since "7 days ago" -p err --no-pager -n 20 2>/dev/null
uptime 2>/dev/null
```

### Compromise Audit
```bash
# 1. Suspicious processes
ps aux --sort=-%cpu | head -30
ps aux | grep -E 'bash -i|/dev/tcp|/dev/udp|ncat|nc -e' | grep -v grep
ps aux | grep -E '^.* /tmp/|^.* /dev/shm' | grep -v grep

# 2. Unauthorized SSH keys & users
cat /etc/passwd | awk -F: '$3 == 0'
for user in $(ls /home/); do ls -la /home/$user/.ssh/ 2>/dev/null; done
sudo ls -la /root/.ssh/ 2>/dev/null
sudo cat /root/.ssh/authorized_keys 2>/dev/null || echo "no root keys"

# 3. Cron jobs
for user in $(cut -d: -f1 /etc/passwd); do crontab -u "$user" -l 2>/dev/null; done
ls -la /etc/cron.d/ /etc/cron.daily/ /etc/cron.hourly/

# 4. Outbound network connections (anomalous destinations)
ss -tupn | grep ESTAB | grep -v "127.0.0.1\|::1\|192.168.\|10\.\|100\.\(64\|115\)\|:41641"

# 5. Kernel modules (rootkit check)
lsmod | grep -vE 'nvidia|drm|nls|ext4|overlay|bridge|br_netfilter|veth|ip_tables|nf_|x_tables'

# 6. Security-sensitive files
find /etc -perm -o+w -type f 2>/dev/null
find / -perm -4000 -type f 2>/dev/null
ls -la /dev/shm/ 2>/dev/null
ls -la /var/run/docker.sock 2>/dev/null

# 7. Recently modified system binaries
find /usr/bin /usr/sbin /usr/local/bin -mtime -7 -type f 2>/dev/null | head -20

# 8. Authentication logs
journalctl -u ssh.service -n 30 --no-pager 2>/dev/null | grep -E "Failed|Accepted|Invalid"
last -20 2>/dev/null
lastb -20 2>/dev/null

# 9. DNS check
cat /etc/resolv.conf 2>/dev/null
cat /etc/hosts 2>/dev/null

# 10. Docker container audit
docker ps -a --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"
```

## Common Cleanup Targets Reference

| Target | Space | Risk | Command |
|--------|-------|------|---------|
| Docker dangling images | 1-7 GB | None | `docker image prune -f` |
| Unused Docker images | 300 MB-7 GB | None (verified) | `docker rmi <name>` |
| Docker build cache | 1-3 GB | Low | `docker builder prune -f` |
| Old kernels | 200-500 MB | None | `sudo apt-get autoremove -y` |
| Apt cache | 1-2 GB | None | `sudo apt-get clean` |
| /tmp old files | 100 MB-1 GB | Low | `sudo rm -rf /tmp/<name>` |
| uv cache | 1-20 GB | None | `uv cache clean` |
| Go build cache | 100-500 MB | None | `rm -rf ~/.cache/go-build/*` |
| pip cache | 100-500 MB | None | `pip cache purge` |
| Arch pacman cache | 5-9 GB | None | `sudo paccache -r -k 2` |
| Arch AUR cache | 1-3 GB | None | `yay -Scc` |
| Arch orphan deps | 10-500 MB | None | `sudo pacman -Rns $(pacman -Qdtq)` |
| Arch old snapper snapshots | 5+ GB | Low | `sudo snapper -c root delete <range>` |
| Gradle cache | 3-7 GB | Low | `find ~/.gradle/caches/ -name "*.jar" -atime +30 -delete` |
| NPM cache | 1-2 GB | None | `npm cache clean --force` |

## Arch/CachyOS Cleanup
```bash
sudo paccache -r -k 2
sudo paccache -ruk0
yay -Scc --noconfirm
sudo pacman -Rns $(pacman -Qdtq) --noconfirm
sudo snapper -c root delete <range>
```

## Coding Agent Tool Removal Reference

| Tool | Binary | Package manager | Config to remove | Typical freed space |
|------|--------|-----------------|------------------|-------------------|
| Cursor Agent | `cursor-agent`, `agent`, `cursor` | standalone (`~/.local/share/cursor-agent/`) | `~/.cursor`, `~/.cursor-server` | ~535 MB |
| Claude Code | `claude` | npm `@anthropic-ai/claude-code` | `~/.claude`, `~/.claude.json` | ~9 MB |
| Codex CLI (OpenAI) | `codex` | npm `@openai/codex` | `~/.codex` | ~61 MB |
| Command Code | `command-code`, `commandcode`, `cmd` | npm `command-code` | none | ~5 MB |
| Cursor IDE | `cursor` desktop binary | AUR `cursor-bin` / pacman | `~/.cursor`, `~/.cursor-server` | ~1 GB |

## Boot Recovery Cascade Patterns

| Visible failure (symptom) | Hidden root cause (likely) |
|---|---|
| UFW: `iptables-restore Failed to initialize nf_tables` | `/boot` vfat mount failure (kernel update dropped vfat from initramfs) |
| ZRAM: `dev-zram0.swap` / `dev-zram0.device` dependency failure | Any base filesystem mount failure (cascading) |
| SDDM/display manager fails to start | XORG or Wayland compositor issues + kernel module mismatch |

## Verification
```bash
df -h / | tail -1
docker system df
sysctl vm.swappiness 2>/dev/null || cat /proc/sys/vm/swappiness
```