# Backup Strategy

## 3-2-1 Rule

- **3 copies** of data (original + 2 backups)
- **2 different media** (NVMe + HDD)
- **1 offsite** (cloud or remote server)

## Current State

- `/backup` directory: nearly empty (4 KB total)
- No automated backup scripts
- No rsync cron jobs
- No pg_dump scheduled
- No snapshot tool (razor-backup, borg, etc.)

## Recommended Setup

### 1. Directory Structure

```
/backup/
├── models/          # /models/downloads (72 GB)
├── hermes/          # ~/.hermes/ (2.7 GB)
├── configs/         # /etc/nginx/, /etc/ssh/, llama-swap config
├── db/              # PostgreSQL dumps, Redis dumps
├── projects/        # /home/rurouni/src, /data/projects
└── snapshots/       # System snapshots (optional)
```

### 2. Automated Backups (rsync)

Create `/etc/cron.daily/server-backup`:

```bash
#!/bin/bash
set -e

BACKUP_DIR="/backup"
DATE=$(date +%Y%m%d-%H%M%S)

# Models (read-heavy, large)
rsync -avz --delete /models/downloads/ "$BACKUP_DIR/models/"

# Hermes config and sessions
rsync -avz --delete ~/.hermes/ "$BACKUP_DIR/hermes/"

# System configs
rsync -avz --delete /etc/nginx/ "$BACKUP_DIR/configs/nginx/"
rsync -avz --delete /etc/ssh/ "$BACKUP_DIR/configs/ssh/"
rsync -avz --delete ~/.config/llama-swap/ "$BACKUP_DIR/configs/llama-swap/"

# Docker compose files
rsync -avz --delete /home/rurouni/projects/openwebui/docker-compose.yml "$BACKUP_DIR/configs/openwebui-compose.yml"
rsync -avz --delete /home/rurouni/projects/searxng/docker-compose.yml "$BACKUP_DIR/configs/searxng-compose.yml"

# Projects (if on NVMe, not RAID0)
rsync -avz --delete /home/rurouni/src/ "$BACKUP_DIR/projects/src/"

# Log backup size
du -sh "$BACKUP_DIR"/* | sort -rh
```

Make executable: `sudo chmod +x /etc/cron.daily/server-backup`

### 3. Database Backups

Create `/etc/cron.daily/db-backup`:

```bash
#!/bin/bash
set -e

BACKUP_DIR="/backup/db"
DATE=$(date +%Y%m%d-%H%M%S)

# PostgreSQL
sudo -u postgres pg_dump openwebui > "$BACKUP_DIR/openwebui-$DATE.sql"
sudo -u postgres pg_dump hermes > "$BACKUP_DIR/hermes-$DATE.sql"

# Redis (copy dump file)
cp /var/lib/redis/dump.rdb "$BACKUP_DIR/redis-$DATE.rdb"

# Clean old dumps (keep 7 days)
find "$BACKUP_DIR" -name "*.sql" -mtime +7 -delete
find "$BACKUP_DIR" -name "*.rdb" -mtime +7 -delete
```

Make executable: `sudo chmod +x /etc/cron.daily/db-backup`

### 4. Remote Backup (Optional)

If you have a remote server or cloud storage:

```bash
# Rsync to remote server
rsync -avz --delete /backup/ user@remote-server:/backup/debian-ai/

# Or use rclone for cloud storage
rclone sync /backup/ remote:debian-ai-backup --progress
```

### 5. Verification

Test backup integrity periodically:

```bash
# Check backup size
du -sh /backup/*

# Verify PostgreSQL dump
sudo -u postgres psql -f /backup/db/openwebui-latest.sql openwebui_test

# Verify rsync consistency
rsync -avz --dry-run /models/downloads/ /backup/models/ | grep -c "^"
```

## Pitfalls

- **RAID0 /data is not backed up** — If /data contains project data, it needs separate backup. RAID0 has zero redundancy.
- **Docker volumes are not in /backup** — Docker volumes live in `/var/lib/docker/volumes/`. Back them up separately if they contain important data.
- **Backup size grows** — Without rotation, backups will fill /backup (1.8 TB HDD). Use `--delete` in rsync and clean old files.
- **Backup runs during peak hours** — Schedule backups during off-peak (e.g., 3 AM) to avoid I/O contention with llama-server.
- **No encryption** — Backups are unencrypted. If /backup is on an external drive, consider encryption.
- **Single point of failure** — If /backup HDD fails, you lose all backups. Consider a second backup drive or cloud storage.