# CachyOS Emergency Mode Recovery — Session 2026-06-13

**System:** CachyOS (Arch), Ryzen 3700X, RTX 3070, NVIDIA 610.43, btrfs root (snapper), Limine bootloader.
**EFI partition:** `/boot` on `nvme1n1p1` (Windows dual-boot ESP, vfat).

**Trigger:** User rebooted after Firefox optimization session (106 prefs, MOZ_ENABLE_WAYLAND=1) + partial system update. Dropped to emergency mode. Two separate sessions: first was cleanup/audit (June 6), second was Firefox tweaks (June 13).

## Error transcript (from `journalctl -xb`)

**Visible failures (symptoms — bottom of journalctl):**
- `ufw.service: Failed with result 'exit-code'` — repeated `iptables-restore/1.8.13 (nf_tables) Failed to initialize nf_tables`
- `dev-zram0.swap: Job dev-zram0.swap/start failed` — cascade
- `systemd-zram-setup@zram0.service: Job failed` — cascade

**Root cause (earliest failure, top of journalctl):**
```
mount: /boot: unknown filesystem type 'vfat'
boot.mount: Mount process exited, code=exited, status=32
Failed to mount /boot.
```

## Key lesson: read journalctl from the TOP

The bottom of `journalctl -xb` shows the most recent failures — UFW and ZRAM. The actual root cause (`/boot` vfat mount failure) was at the top. Always scroll up past the visible failures to find the first error in the boot chain.

## Quick fix (get the system booting)

From emergency mode (enter root password):

```bash
# Option A: Comment out /boot in fstab (fastest)
sed -i '/\/boot/s/^/#/' /etc/fstab
exit
```

```bash
# Option B: Mask cascade failures (if fstab isn't the issue)
systemctl mask dev-zram0.swap
systemctl mask systemd-zram-setup@zram0.service
exit
```

**Note:** `systemctl disable` does NOT work on ZRAM template units — they lack `[Install]` section. Must use `mask`.

## Blank screen after exit

If the machine reaches a blank screen with a blinking dash after `exit` from emergency mode:

1. Press **Ctrl+Alt+F2** (or F3-F6) for a TTY
2. Log in with root password
3. Check display manager: `systemctl status sddm`
4. The machine may be unreachable on LAN (`ssh: No route to host`) and Tailscale shows `offline`

## Snapshot recovery (what worked)

The user boot-selected a btrfs snapshot from the Limine boot menu. After booting from snapshot:

- `df -h /` shows `overlay` — normal for snapshot boot
- `/boot` fstab entry is restored to original (snapshot reverts root, not `/boot`)
- `pacman -Qu` shows many pending updates (the updates were rolled back)
- `pacman -Syu` may abort with **stale db.lck** → `sudo rm -f /var/lib/pacman/db.lck`

## Root cause analysis

The `vfat` kernel module (`/lib/modules/*/kernel/fs/fat/vfat.ko.zst`) was not included in the initramfs generated after a kernel update. The `autodetect` mkinitcpio hook only includes modules for hardware detected at build time — vfat was missed despite `/boot` being a vfat partition.

**Why it happens:** The kernel update triggers mkinitcpio (via pacman hook), which runs `autodetect`. If vfat isn't actively detected as needed (e.g., `/boot` is already mounted at build time and the detection heuristic misses it), it's excluded.

## Permanent fix (Limine — CachyOS default)

CachyOS uses **Limine** bootloader — NOT systemd-boot or GRUB. The initramfs is rebuilt with `limine-mkinitcpio`, not `mkinitcpio -P`.

```bash
# Step 1: Add vfat to MODULES array
sudo sed -i 's/^MODULES=()$/MODULES=(vfat)/' /etc/mkinitcpio.conf

# Step 2: Rebuild via the correct Limine variant
sudo limine-mkinitcpio

# Step 3: Verify vfat is in the new image
ls /boot/*/linux-cachyos/ 2>/dev/null
lsinitcpio /boot/<uuid>/linux-cachyos/initramfs-linux-cachyos | grep vfat
# Expected: usr/lib/modules/<version>-cachyos/kernel/fs/fat/vfat.ko.zst
```

**Wrong command pitfall:**
```
$ sudo mkinitcpio -P
==> ERROR: No presets found in /etc/mkinitcpio.d
==> WARNING: This does not update Limine boot entries.
    Use 'limine-mkinitcpio' or 'limine-update' instead.
```

## Full recovery workflow (this session)

### Phase 1: Boot from snapshot

1. Booted into snapper snapshot → system up
2. Added `MODULES=(vfat)` to `/etc/mkinitcpio.conf`
3. Rebuilt: `sudo limine-mkinitcpio`
4. Verified `vfat.ko.zst` in new initramfs (path: `/boot/<uuid>/linux-cachyos/initramfs-linux-cachyos`)
5. Removed stale `db.lck`: `sudo rm -f /var/lib/pacman/db.lck`
6. Ran `sudo pacman -Syu` — 98 packages updated, kernel went from 7.0.11-1 → 7.0.12-1

### Phase 2: Reboot — black screen, NVIDIA driver missing

After reboot, machine booted but KWin fell back to **software rendering**. Root cause: **NVIDIA DKMS modules weren't built** for the new kernel.

**NVIDIA DKMS module name pitfall:**
- Installed package: `nvidia-open-dkms 610.43.02-3`
- DKMS registered as: `nvidia/610.43.02` (not `nvidia-open`)
- Source directory: `/usr/src/nvidia-610.43.02/` (not `/usr/src/nvidia-open-610.43.02/`)
- If you try `dkms install -m nvidia-open ...`, it fails with `Error! Could not find module source directory.`

**Fix:**
```bash
# Build with the correct module name (nvidia, not nvidia-open)
sudo dkms install -m nvidia -v 610.43.02 -k 7.0.12-1-cachyos

# Load all NVIDIA modules
sudo modprobe nvidia
sudo modprobe nvidia_uvm
sudo modprobe nvidia_drm
sudo modprobe nvidia_modeset

# Ensure nvidia_uvm persists across boots
echo "nvidia_uvm" | sudo tee /etc/modules-load.d/nvidia-uvm.conf
```

### Phase 3: Post-fix cleanup

Three things were broken by the emergency mode workarounds:

1. **ZRAM masked** — `mask` sticks across reboots. Unmask:
   ```bash
   sudo systemctl unmask systemd-zram-setup@zram0.service
   sudo systemctl unmask dev-zram0.swap
   sudo systemctl start systemd-zram-setup@zram0
   ```

2. **UFW inactive** — was disabled during emergency mode. Restore:
   ```bash
   echo "y" | sudo ufw enable
   ```

3. **ai-router.service spam** — unrelated service repeatedly failing. Disable:
   ```bash
   sudo systemctl disable --now ai-router.service
   ```

### Phase 4: Post-update verification (11 checks)

After completing update + NVIDIA DKMS build + cleanup:

```bash
# 1. Kernel landed
uname -r

# 2. Initramfs has vfat and nvidia
lsinitcpio /boot/<uuid>/linux-cachyos/initramfs-linux-cachyos | grep -E "vfat|nvidia"

# 3. NVIDIA driver loaded
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv,noheader

# 4. Failed services — expect 0
systemctl --failed --no-pager --no-legend

# 5. Critical services active
for svc in sddm NetworkManager ufw systemd-oomd fail2ban fstrim.timer ananicy-cpp; do echo "$svc: $(systemctl is-active "$svc")"; done

# 6. ZRAM swap active
swapon --show

# 7. Package integrity — 0 warnings
pacman -Qk 2>&1 | grep -c "WARNING\|MISSING"

# 8. No orphans accumulating
pacman -Qdt | wc -l

# 9. No pending updates
pacman -Qu | wc -l

# 10. Disk health
sudo smartctl -A /dev/nvme0n1 2>/dev/null | grep -E "SMART|Percentage Used|Temperature"
sudo btrfs device stats /

# 11. Boot params correct
cat /proc/cmdline
```