---
name: linux-desktop-troubleshooting
description: >-
  Diagnose and fix Linux desktop display failures (black screen, DM crash, login loop) on
  remote machines — especially rolling-release distros (Arch/CachyOS) with NVIDIA GPUs.
  Handles the bootstrap problem: SSH key not authorized → work through TTY commands.
triggers:
  - user reports black screen, login crash, display manager failure on Linux desktop
  - user reports "black screen after login" or "putting in password and black screen"
  - user reports blurry, pixelated, messy, or weird-looking text on Linux desktop
  - need to diagnose GPU/display remotely but SSH not available
  - user on TTY (Ctrl+Alt+F2/F3) after GUI failure
  - NVIDIA driver update broke display on Arch-based distro
  - CachyOS snapper restore fails with "FAT32 boot partition is not mounted at '/boot'" or pre-hook 10-limine-reset-enroll error
  - CachyOS snapshot rollback blocked by boot hook failure
toolsets:
  - terminal
  - file
---

# Linux Desktop Troubleshooting

## Bootstrap Checklist (SSH Unavailable)

See `references/ssh-bootstrap.md` for the exact pattern — saves 3-4 timeout retries.

When the target machine's SSH key isn't authorized and you need the user to run commands:

1. **Check your side first** — Run `ssh-add -l` on your machine. If "The agent has no identities", load your key: `ssh-add ~/.ssh/id_ed25519`
2. **Confirm TTY access** — Ctrl+Alt+F2/F3, text login. Run `whoami` — if root, use absolute paths for the target user's home.
2. **Verify networking** — `ip a | grep -E "192\.168|100\.|state UP"`
3. **Start SSH if possible** — `sudo systemctl start sshd` (start, don't enable until test passes)
4. **Confirm port open** — `ss -tlnp | grep 22`
5. **Add your SSH key** — two options:

   **Option A** (direct, short key): `echo "ssh-ed25519 AAAA..." >> ~/.ssh/authorized_keys; chmod 600 ~/.ssh/authorized_keys`
   **If user is root**, use absolute path: `echo "ssh-ed25519 AAAA..." >> /home/raymond/.ssh/authorized_keys; chmod 600 /home/raymond/.ssh/authorized_keys; chown raymond:raymond /home/raymond/.ssh/authorized_keys`

   **Option B** (HTTP server — NO typing of long key): Serve your pubkey via Python on your end:
   ```bash
   # On YOUR machine (the one SSHing out):
   echo "ssh-ed25519 AAAA..." > /tmp/pubkey.txt
   cd /tmp && python3 -m http.server 18900 --bind 0.0.0.0
   # Verify locally first:
   curl -s http://127.0.0.1:18900/pubkey.txt
   ```
   Then have user run:
   ```bash
   curl -s http://<YOUR_IP>:18900/pubkey.txt >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys
   ```
   **If user is root**, replace `~` with the target user's absolute path (e.g. `/home/raymond/.ssh/authorized_keys`), and add `chown` to fix ownership.

6. **Verify** — Have the user confirm: `tail -3 ~/.ssh/authorized_keys` (or absolute path if they're root). The line must start with `ssh-ed25519` and be on one line — don't retry SSH without confirming.
7. **Test** — SSH from your side. If it still hangs, check `ssh -vvv` — the hang-at-auth pattern confirms authorized_keys issue, not network.

### Last Resort: Reverse SSH Tunnel
If the target machine can SSH out to you but you can't SSH in (key auth keeps failing), have the user establish a reverse tunnel:

```bash
# User runs on target TTY (keeps running):
ssh -R 2222:localhost:22 <YOUR_USER>@<YOUR_IP>
```

Then from your machine:
```bash
ssh -p 2222 raymond@127.0.0.1
```

This works because it tunnels through the outgoing connection which already authenticated to your side.

**⚠️ Circular key trap:** The command above will **fail immediately** if the target machine's SSH key isn't authorized in `<YOUR_USER>`'s `~/.ssh/authorized_keys` on your side. Same bootstrap problem, reversed. Before recommending this:
1. Check if the target has a public key: have the user run `cat ~/.ssh/id_ed25519.pub`
2. Add it to your machine's `~/.ssh/authorized_keys`
3. Then the reverse tunnel command will stay open

If the target has no SSH keypair at all (`No such file or directory`), skip this approach entirely and fall back to the HTTP server method (Option B above).

## Target-Host Verification (before any diagnosis)

When the user names a different machine than the current session host (for example "the CachyOS box" while you're running on Debian), **verify and audit the target host first** instead of inferring from the local machine.

1. Confirm the target by name/IP/OS from prior context.
2. If SSH works, audit the target directly before offering fixes.
3. Do **not** conclude "missing PipeWire/PulseAudio/ALSA" from the wrong host.
4. If the symptom is app-specific audio, check the live app stream state before package assumptions.

## Firefox / PipeWire No-Sound Triage (remote desktop)

For Firefox/YouTube no-audio on KDE/PipeWire systems, check the target host in this order:

```bash
# Core stack present + running
pacman -Q pipewire pipewire-pulse pipewire-alsa wireplumber alsa-utils 2>&1 || true
systemctl --user --no-pager --full status pipewire.service pipewire-pulse.service wireplumber.service 2>&1 || true
pactl info 2>&1 || true
wpctl status 2>&1 || true

# App-specific stream state while Firefox is actively playing audio
pactl list sink-inputs | sed -n '/Sink Input #/,+35p' | grep -E 'Sink Input #|Sink:|Mute:|Volume:|application.name|media.name|node.name'
```

**Key rule:** if PipeWire is healthy and Firefox appears under `sink-inputs`, inspect `Mute:` on the Firefox stream before recommending package installs or service changes.

**Fast fix when only Firefox is muted:**
```bash
pactl set-sink-input-mute <sink-input-id> 0
```

Then re-check:
```bash
pactl list sink-inputs | sed -n '/Sink Input #<id>/,+20p' | grep -E 'Mute:|Volume:|application.name|media.name'
```

## Font Rendering — Blurry/Pixelated Text

When the user reports text looks "weird", "messy", "blurry", or "pixelated" on a KDE desktop (especially Arch/CachyOS with NVIDIA + mixed DPI monitors), the most common root cause is **anti-aliasing disabled in KDE config**, not missing fonts or drivers.

### Quick Check (SSH from TTY)

```bash
# Check Xft settings in KDE config
grep "Xft" ~/.config/kdeglobals
```

**Expected output** (all four lines present):
```
XftAntialias=true
XftRGBA=rgb
XftHintStyle=hintslight
XftSubPixel=rgb
```

If `XftAntialias=false` → **that's the root cause**. If any line is missing, the default may be wrong.

### Diagnosis Checklist

```bash
# 1. Check anti-aliasing and hinting state
grep "Xft" ~/.config/kdeglobals

# 2. Check freetype2 version (vanilla Arch package has ClearType built-in since 2.13+)
pacman -Q freetype2

# 3. Check KDE screen config (scale, resolution, DPI)
cat ~/.local/share/kscreen/outputs/* 2>/dev/null

# 4. Check if FREETYPE_PROPERTIES is set
echo "${FREETYPE_PROPERTIES:-unset}"

# 5. Check current font and size
grep "^font=" ~/.config/kdeglobals

# 6. Check KScreen monitor config for mixed DPI (4K + 1080p is a common pain point)
cat ~/.local/share/kscreen/*.json 2>/dev/null | python3 -c "
import sys, json
d = json.load(sys.stdin)
if isinstance(d, dict) and 'outputs' in d:
    for o in d['outputs']:
        print(f\"{o.get('name','?'):12} {o.get('resolution','?'):12} scale={o.get('scale',1)}\")"
```

### Fix — Anti-Aliasing Disabled

```bash
sed -i 's/XftAntialias=false/XftAntialias=true/' ~/.config/kdeglobals

# Ensure RGB subpixel rendering is set
grep -q 'XftRGBA' ~/.config/kdeglobals \
  && sed -i 's/XftRGBA=.*/XftRGBA=rgb/' ~/.config/kdeglobals \
  || sed -i '/^\[General\]/a XftRGBA=rgb' ~/.config/kdeglobals

grep -q 'XftSubPixel' ~/.config/kdeglobals \
  && sed -i 's/XftSubPixel=.*/XftSubPixel=rgb/' ~/.config/kdeglobals \
  || sed -i '/^XftRGBA/a XftSubPixel=rgb' ~/.config/kdeglobals
```

### Fix — Optimal freetype2 Tuning

Create `/etc/environment.d/99-font-rendering.conf`:

```bash
sudo mkdir -p /etc/environment.d
sudo tee /etc/environment.d/99-font-rendering.conf << 'EOF'
FREETYPE_PROPERTIES="cff:no-stem-darkening=1 autofitter:warping=1"
EOF
```

- `cff:no-stem-darkening=1` — removes muddy over-darkening on CFF/OpenType fonts (Inter, Noto, etc.)
- `autofitter:warping=1` — crisper edges, closer to Windows ClearType

### Fix — Local fontconfig Config

```bash
mkdir -p ~/.config/fontconfig
cat > ~/.config/fontconfig/fonts.conf << 'XML'
<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "urn:fontconfig:fonts.dtd">
<fontconfig>
  <match target="pattern">
    <edit name="dpi" mode="assign"><double>96</double></edit>
  </match>
  <match target="font">
    <edit name="antialias" mode="assign"><bool>true</bool></edit>
    <edit name="hinting" mode="assign"><bool>true</bool></edit>
    <edit name="hintstyle" mode="assign"><const>hintslight</const></edit>
    <edit name="rgba" mode="assign"><const>rgb</const></edit>
    <edit name="lcdfilter" mode="assign"><const>lcddefault</const></edit>
  </match>
  <selectfont>
    <rejectfont>
      <pattern>
        <patelt name="fontformat"><string>FIXED</string></patelt>
      </pattern>
    </rejectfont>
  </selectfont>
</fontconfig>
XML
```

### Recipe: Recognize Mixed-DPI Pain

When the user has both a 4K monitor and 1080p monitors (common with NVIDIA multi-monitor):

1. **Check the KScreen outputs** — `cat ~/.local/share/kscreen/outputs/*` shows resolution and scale per monitor
2. **No scaling + 4K TV at couch distance** — text is tiny. Bump font: `sed -i 's/^font=Inter,10,/font=Inter,11,/' ~/.config/kdeglobals`
3. **Inter 11pt** is a good starting size for mixed 4K/1080p setups
4. No font scaling needed at this size — KDE scale=1.0 works fine with larger base font

### Apply Without Logout

```bash
kwin_x11 --replace
```

Or just **log out and back in**.

### Verification

```bash
grep "Xft" ~/.config/kdeglobals
# Should show: antialias=true, rgba=rgb, subpixel=rgb, hintslight

cat /etc/environment.d/99-font-rendering.conf
# Should show FREETYPE_PROPERTIES set

cat ~/.config/fontconfig/fonts.conf
# Should show anti-alias + hinting + lcdfilter config
```

For a detailed recipe with all config files and exact commands from a real CachyOS fix session, see `references/font-rendering.md` in this skill's directory.

## Black Screen After Login — Diagnosis Order

Run these in order, have user output piped back:

### A. Display Manager Status
```bash
# Check what DM is active
cat /etc/systemd/system/display-manager.service 2>/dev/null
# Check DM process
ps aux | grep -E "(sddm|gdm|lightdm|greetd)" | grep -v grep
# Check DM log
cat /var/log/sddm.log 2>/dev/null | tail -40
```

### B. GPU + Driver Info
```bash
nvidia-smi
# Check for no running processes = display server crashed
```

### C. Recent Package Changes (most common cause on rolling distros)
```bash
cat /var/log/pacman.log | grep -E "installed|upgraded" | grep -E "nvidia|linux|kde|plasma|sddm" | tail -20
```

## Common Fixes (ordered by likelihood on Arch/CachyOS)

### 0. Firefox/YouTube Has No Audio on PipeWire (verify stream state before package assumptions)
If the user names a different machine (e.g. Debian host vs CachyOS desktop), **audit that target directly first**. Do not infer CachyOS audio state from the current local machine.

For browser-audio complaints on KDE/CachyOS, **do not assume missing packages**. First prove whether the issue is the audio stack, the selected sink, or a per-app mute on the Firefox stream.

**Audit in this order:**
```bash
# 1) Verify PipeWire/Pulse bridge is actually present and running
pacman -Q pipewire pipewire-pulse pipewire-alsa wireplumber alsa-utils
systemctl --user --no-pager --full status pipewire.service pipewire-pulse.service wireplumber.service
pactl info

# 2) Confirm the active default sink (HDMI TV vs headset matters)
pactl info | grep "Default Sink"
pactl list short sinks
wpctl status

# 3) While a YouTube video is playing in Firefox, inspect Firefox's sink-input
pactl list sink-inputs | sed -n '/Sink Input #/,+35p' | grep -E 'Sink Input #|Sink:|Mute:|Volume:|application.name|media.name|node.name'
# If you need the full Firefox block reliably:
python3 - <<'PY'
import subprocess,re
out=subprocess.check_output(['pactl','list','sink-inputs'], text=True, errors='replace')
for b in re.split(r'(?m)^Sink Input #', out):
    if 'application.name = "Firefox"' in b:
        print('Sink Input #'+b.splitlines()[0])
        for line in b.splitlines()[1:]:
            if any(k in line for k in ['Sink:','Mute:','Volume:','application.name','media.name','node.name']):
                print(line)
PY
```

**Interpretation:**
- `Server Name: PulseAudio (on PipeWire ...)` + Firefox stream present = stack is healthy enough for app playback.
- Firefox stream on the expected sink with `Mute: yes` = **root cause is per-app mute**, not missing packages.
- Default sink on the wrong device (e.g. HDMI TV instead of headset) = routing problem, not Firefox.

**Immediate fix for per-app mute:**
```bash
pactl set-sink-input-mute <sink_input_id> 0
```

**Likely trigger when mute appears "randomly":** PipeWire/WirePlumber restart or crash during the session. Check:
```bash
journalctl --user --since "2 hours ago" --no-pager | grep -Ei '(firefox|pipewire|wireplumber|mute|volume|sink-input|stream)'
```
If the journal shows PipeWire/WirePlumber stopping, Firefox `cubeb` audio errors, or a WirePlumber crash (`status=6/ABRT`) near the failure window, treat that as the likely cause of the bad stream state.

See `references/firefox-pipewire-audio.md` for a concise remote-audit checklist and evidence pattern.

### 1. NVIDIA Driver + Kernel Mismatch (🔴 most common)
Rolling update installed new kernel but nvidia-dkms didn't rebuild:
```bash
# Check what driver version is active vs installed
nvidia-smi | grep "Driver Version"
pacman -Qs nvidia | grep installed
# Rebuild
sudo pacman -S linux-cachyos nvidia-dkms
# Or fallback to LTS kernel
sudo pacman -S linux-cachyos-lts nvidia-lts
```
**Boot fallback kernel from GRUB** first to confirm.

### 2. NVIDIA 610 Driver Branch (🔴 common)
Branch 610.x is beta/dev and frequently breaks KDE/Wayland compositors.
```bash
# Check current version
nvidia-smi --query-gpu=driver_version --format=csv,noheader
# Downgrade to 570/565 branch via pacman cache or mirror rollback
```
### 3. SDDM Wayland Crash → Force X11

```bash
echo "[Autologin]
Session=plasma-x11" | sudo tee /etc/sddm.conf.d/kde_settings.conf
```
Or edit `/etc/sddm.conf`:
```ini
[General]
DisplayServer=x11
```

**⚠️ Also check `RememberLastSession` in the global `/etc/sddm.conf`.** Even with `DefaultSession=plasma-x11` set, if `/etc/sddm.conf` has `RememberLastSession=true`, SDDM overrides the default with whatever the user last selected (e.g. Kodi). Fix:

```bash
sed -i 's/RememberLastSession=true/RememberLastSession=false/' /etc/sddm.conf
# Also clear persisted session state:
rm -f /var/lib/sddm/.last_session /home/<user>/.config/sddm_state.conf
```

### 3b. Missing Plasma X11 Session File
Even when `plasma-workspace` and `plasma-desktop` are installed, `/usr/share/xsessions/plasma.desktop` can be **absent** — the package owns the directory but the session file itself may be missing. This means SDDM won't show Plasma as an X11 option (only Kodi or other surviving sessions).

**Check:**
```bash
ls /usr/share/xsessions/
# If plasma.desktop is missing but startplasma-x11 exists →
```

**Fix — create the .desktop file:**
```bash
cat > /usr/share/xsessions/plasma.desktop << 'EOF'
[Desktop Entry]
Type=Application
Name=Plasma (X11)
Comment=Plasma Desktop Workspace
Exec=/usr/bin/startplasma-x11
TryExec=/usr/bin/startplasma-x11
DesktopNames=KDE
X-KDE-PluginInfo-Name=plasma
EOF
```

**Always verify the binary exists first:** `which startplasma-x11`

Available as `templates/plasma-x11.desktop` in this skill's directory for quick copy.

### 3c. User-Level KDE Config Corruption (one user works, another doesn't)
If root logs in fine but raymond gets black screen (or vice versa), the broken user has corrupted KDE config from a previous Wayland session. System-wide driver/DM is fine.

**Fix — clear per-user KDE config:**
```bash
rm -f ~/.config/kdeglobals ~/.config/plasma-org.kde.plasma.desktop-appletsrc ~/.config/kwinrc ~/.config/plasmashellrc
```
KDE regenerates clean defaults on next login. If the damage is extensive, back up and remove the entire `.config` dir: `mv ~/.config ~/.config.bak`

### 4. No Display Server Running At All
If `nvidia-smi` shows no GPU processes → DM is crashing before it launches.
- Try manual start from TTY: `export DISPLAY=:0 && startplasma-x11` (Ctrl+C to return)
- Check `dmesg | grep -i nvidia | tail -10` for kernel oops

## Working from a Live Environment / Btrfs Snapshot

If the user booted into a rescue/live environment (overlay filesystem) because the normal system won't boot, you can't just fix `/` directly — changes go to the overlay and are lost on reboot.

### Identify the Situation

```bash
# Check if running in overlay (live/rescue)
mount | grep overlay
# If present, lowerdir points to read-only root, upperdir is ephemeral
# Example: overlay on / type overlay (rw,lowerdir=/sysroot,upperdir=/cowspace_JoH/upper,...)

# Check what system services are active
systemctl status display-manager --no-pager -n 5
```

### Find and Mount the Real Root

```bash
# List block devices to find the root partition
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT

# List Btrfs subvolumes (CachyOS standard layout)
btrfs subvolume list /home

# Typical CachyOS layout:
# ID 256 → @          (root filesystem)
# ID 257 → @home      (home directories)
# ID 258 → @root      (root user home)
# ID 260 → @cache     (pacman cache)
# ID 265 → @/.snapshots/  (snapper snapshots)

# Mount the real root @ subvol
mount -o subvol=@ /dev/nvme1n1p2 /mnt/root  # adjust device + subvol ID as needed
```

### Apply System-Level Fixes to the Real Root

All system config goes to `/mnt/root/...`, NOT the live environment's `/`:

```bash
# Example: add SDDM config
mkdir -p /mnt/root/etc/sddm.conf.d
cat > /mnt/root/etc/sddm.conf.d/kde-settings.conf << 'EOF'
[Users]
DefaultSession=plasma-x11
[Theme]
Current=breeze
EOF

# Example: create missing Plasma X11 session file
mkdir -p /mnt/root/usr/share/xsessions
cat > /mnt/root/usr/share/xsessions/plasma.desktop << 'EOF'
[Desktop Entry]
Type=Application
Name=Plasma (X11)
Comment=Plasma Desktop Workspace
Exec=/usr/bin/startplasma-x11
TryExec=/usr/bin/startplasma-x11
DesktopNames=KDE
X-KDE-PluginInfo-Name=plasma
EOF

# Example: enable root SSH in real root
sed -i 's/^PermitRootLogin no/PermitRootLogin prohibit-password/' /mnt/root/etc/ssh/sshd_config
```

**Important:** The `@home` subvol is typically already mounted at `/home/` in the live environment — user-specific fixes (fish config, KDE config, SSH keys) go directly to `/home/<user>/...` and persist because that mount is the real subvol.

### Verify Fixes Applied

```bash
cat /mnt/root/etc/sddm.conf.d/kde-settings.conf     # should show plasma-x11
cat /mnt/root/usr/share/xsessions/plasma.desktop     # should show Exec=startplasma-x11
grep PermitRootLogin /mnt/root/etc/ssh/sshd_config | grep -v '#'
```

### Set Default Btrfs Subvolume

After applying fixes to the `@` subvol, ensure the system boots from it by default:

```bash
btrfs subvolume set-default 256 /mnt/root   # 256 = subvol ID of @
```

Without this, the kernel cmdline's `rootflags=subvol=/@` may work anyway, but setting the default ensures consistency with the bootloader entry.

### Fix the Bootloader (Limine)

CachyOS uses **Limine** bootloader, not GRUB. Config is at `/boot/limine.conf` on the ESP partition:

```bash
# Mount the ESP partition first
mount /dev/nvme1n1p1 /mnt/root/boot   # adjust partition

# Check current default
grep default_entry /mnt/root/boot/limine.conf

# Common issue: default_entry: 2 points to the snapshots section,
# causing the system to boot into an old snapshot (which may have Kodi
# as the only X11 session, or black screen from a previous broken state)
# Fix: set to 0 (first normal kernel entry)
sed -i 's/default_entry: 2/default_entry: 0/' /mnt/root/boot/limine.conf
```

Limine entries are 0-indexed. Typical layout:
- Entry 0: linux-cachyos (main kernel)
- Entry 1: linux-cachyos-lts (LTS fallback)
- Entry 2: Snapshots section (default in many CachyOS installs — trips up users)

### Snapper `undochange` Failure in Overlay/Chroot

When you're in a live/rescue environment (overlay root) or chroot, `snapper undochange` may fail with:

```
IO Error (query default id failed, subvolume is not a btrfs subvolume).
```

**Why:** Snapper needs direct btrfs ioctl access to the top-level subvolume. Inside a chroot where `/` is a mounted btrfs subvol (e.g. `subvol=/@`), snapper can't find the btrfs root. It also fails from the overlay filesystem in a live/rescue session.

**Workaround — Direct btrfs subvolume swap:**

Instead of fighting snapper, replace the `@` subvolume directly:

```bash
# 1. Mount the btrfs top-level (subvolid=5)
sudo mount -o subvolid=5 /dev/nvme0n1p2 /mnt/btrfs

# 2. Verify the snapshots are there
sudo ls /mnt/btrfs/@/.snapshots/

# 3. Backup the current @ (saves ~256GB, can be deleted later)
sudo mv /mnt/btrfs/@ /mnt/btrfs/@.broken

# 4. Create a writable snapshot of the working snapshot as new @
#    (replace 164 with your target snapshot number)
sudo btrfs subvolume snapshot /mnt/btrfs/@.broken/.snapshots/164/snapshot /mnt/btrfs/@

# 5. Fix fstab in new @ (snapshot carries old broken fstab)
sudo sed -i 's|^#UUID=F594-D99E|UUID=F594-D99E|g' /mnt/btrfs/@/etc/fstab

# 6. Mount ESP and verify
sudo mkdir -p /mnt/btrfs/@/boot
sudo mount /dev/nvme0n1p1 /mnt/btrfs/@/boot
ls /mnt/btrfs/@/boot/limine.conf
```

**Caveats:**
- **Snapshot history is lost in the new `@`** — The `.snapshots` directory is a nested btrfs subvolume (typically ID 265+). When you create a snapshot of a snapshot, nested subvolumes become empty directories. The old snapshots remain under `@.broken/.snapshots/` — backup `@.broken` if you need them later.
- **@.broken can be deleted:** `sudo btrfs subvolume delete /mnt/btrfs/@.broken` — do this only after confirming the new `@` boots correctly.
- **System will boot into the new `@` automatically** if kernel cmdline uses `rootflags=subvol=/@` (standard CachyOS). No `btrfs subvolume set-default` needed.

### Snapper Restore Hook Failure (10-limine-reset-enroll)

CachyOS installs a pre-hook at `/etc/boot/hooks/pre.d/10-limine-reset-enroll` that runs before snapper restore/undochange operations. It checks:
1. Is a FAT32 partition mounted at `/boot`?
2. Does `/boot/limine.conf` exist?

**Error signature:**
```
ERROR: FAT32 boot partition is not mounted at '/boot'.
WARNING: pre hook failed (exit code 1): /etc/boot/hooks/pre.d/10-limine-reset-enroll
The file: /boot/limine.conf is not found.
```

**Why:** CachyOS mounts the ESP **directly at `/boot`** (not `/boot/efi`). The hook checks for this to reset secure boot enrollment keys (via `sbctl`) during a snapshot rollback. If the ESP isn't at `/boot`, the hook fails and blocks the restore.

**Fixes (in order of preference):**

**Option A — Mount ESP at /boot (correct):**
```bash
# Find the ESP partition
lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT | grep vfat
# Mount it at /boot
mount /dev/nvme0n1p1 /boot
# Verify
ls /boot/limine.conf
```

**Option B — Bind mount if ESP is at /boot/efi:**
```bash
# If ESP is already mounted at /boot/efi, bind-mount it to /boot
mount --bind /boot/efi /boot
```
This works when the live environment mounts the ESP at `/boot/efi` (common in some rescue images) but the hook expects it at `/boot`. The bind mount makes the same files visible at both paths.

**Option C — Skip the hook (quick workaround):**
```bash
chmod -x /etc/boot/hooks/pre.d/10-limine-reset-enroll
# Run the restore, then re-enable:
chmod +x /etc/boot/hooks/pre.d/10-limine-reset-enroll
```

**Permanent fix — Fix fstab so ESP always mounts at /boot:**
```
/dev/nvme0n1p1  /boot  vfat  umask=0077  0  1
```
Do this from `arch-chroot` after mounting the real root.

### Run Commands in the Real Root (chroot)

For `systemctl` commands (e.g. mask/unmask services):

```bash
# Bind-mount virtual filesystems
mount --bind /dev /mnt/root/dev
mount --bind /proc /mnt/root/proc
mount --bind /sys /mnt/root/sys

# Chroot and run command
chroot /mnt/root systemctl mask kodi   # prevent Kodi from interfering
```

### Cleanup

```bash
umount /mnt/root/dev /mnt/root/proc /mnt/root/sys 2>/dev/null
umount /mnt/root/boot 2>/dev/null
umount /mnt/root 2>/dev/null
```

For detailed CachyOS Btrfs subvolume layout, Limine bootloader config format, and live-environment mount procedures, see `references/cachyos-btrfs-limine.md` in this skill's directory.

## Pitfalls

- **`RememberLastSession=true` overrides `DefaultSession` in SDDM** — Even when `DefaultSession=plasma-x11` is set in `/etc/sddm.conf.d/kde-settings.conf`, the global `/etc/sddm.conf` has `RememberLastSession=true` by default on many Arch/CachyOS installs. If the user last selected Kodi (or any non-Plasma session), SDDM remembers that and boots into it on login, ignoring the DefaultSession directive. **Fix:** `sed -i 's/RememberLastSession=true/RememberLastSession=false/' /etc/sddm.conf` — or set it to `false` so DefaultSession is always honored. Also clear any persisted last-session state: `rm -f /var/lib/sddm/.last_session /home/<user>/.config/sddm_state.conf`.
- **PasswordAuthentication no** is default on Arch — only key auth works. If key isn't authorized, SSH hangs at auth phase (even with `-v` showing nothing). Fix: user adds key manually.
- **SSH hangs but ping works + port 22 open** = almost always key auth issue, not network.
- **`systemctl status sddm` returns nothing** when the DM service unit is masked or not found. Always check `/etc/systemd/system/display-manager.service` for the symlink.
- **NVIDIA 610 branch**: as of mid-2026, this is a beta branch. Common issues: Wayland compositor crash, SDDM freeze, blank screen after password. Downgrade to 570 or use LTS kernel.
- **No output from DM log** = DM never initialized. Check if display-manager symlink exists at all.
- **Don't recommend driver rollback without knowing current version first** — some users need 610 for newer GPU support (RTX 50 series).
- **User may push back on typing long SSH keys or "just run these commands"** — the HTTP server key delivery (Option B in Bootstrap) avoids both problems. Try SSH at least 4-5 times with varied approaches before resorting to handing off commands entirely.
- **Always verify the pubkey server works locally first** — Python http.server 404s if started from the wrong directory. Test with `curl -s http://127.0.0.1:<port>/pubkey.txt` before telling the user to fetch.
- **"try now" without verification** — After the user says they ran the key-add command, verify the key format on your end by asking `tail -3 ~/.ssh/authorized_keys` so you can confirm the line is well-formed before retrying SSH.
- **TTY `~` expands to `/root/` when logged in as root** — If the user is root on TTY, `ls -la ~/.ssh/` shows `/root/.ssh/`, NOT `/home/<user>/.ssh/`. Always check with `whoami` first and use absolute paths (e.g. `/home/raymond/.ssh/authorized_keys`).
- **SSH hang with key on disk but not in agent** — The `id_ed25519` file may exist on disk with correct permissions, but `ssh-add -l` shows "The agent has no identities." Run `ssh-add ~/.ssh/id_ed25519` early as a standard step before deep-diving into remote authorized_keys.
- **Fish shell `config.fish` with recursive `fish set -x` bug** — A config.fish ending with `fish set -x PATH ...` spawns a recursive fish-in-fish process on every shell launch, causing SSH to hang at session exec (after auth succeeds). The SSH log shows `session opened` but the shell never starts. Fix: `sed -i '$ s/^fish //' ~/.config/fish/config.fish` to change it to `set -x PATH ...`. Check for this when SSH auth passes but command execution hangs.
- **Kodi package can overwrite xsessions directory** — Installing `kodi` can silently delete existing `.desktop` files from `/usr/share/xsessions/` because `kodi` claims ownership of that directory. Even when `plasma-workspace` and `plasma-desktop` are installed, `plasma.desktop` for X11 may vanish, leaving only `kodi.desktop`. The user then sees only Kodi as a login option (or Kodi boots automatically). Fix is creating the `plasma.desktop` file manually (see section 3b above). Check with `pacman -Qo /usr/share/xsessions/` to see which packages claim ownership.
- **One user breaks, another works (root vs raymond)** — If root can log in to the desktop but raymond gets black screen, it's almost always user-level KDE/Wayland config corruption, not a system-wide driver issue. Fix: clear raymond's KDE config (`rm -f ~/.config/kdeglobals ~/.config/plasma-org.kde.plasma.desktop-appletsrc ~/.config/kwinrc ~/.config/plasmashellrc`). KDE regenerates defaults on next login.
- **Reverse tunnel fails if target key not authorized on your side** — `ssh -R 2222:localhost:22 <YOUR_USER>@<YOUR_IP>` exits immediately if the target's SSH key isn't in your authorized_keys. This is the reverse of the bootstrap problem. Before recommending this, check if the target has a public key (`cat ~/.ssh/id_ed25519.pub`) — if not, generate one, or fall back to serving your key via HTTP instead.
- **`PermitRootLogin` change needs `systemctl restart sshd` to take effect** — `sed`-ing sshd_config alone won't apply until the daemon reloads. Always append `&& systemctl restart sshd` after the sed.
- **Drive identification across live sessions — use MODEL column** — When working from a live/rescue environment, `lsblk` output can vary between boots (device ordering `nvme0n1` vs `nvme1n1` can swap). **Always use `lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT,MODEL`** to identify the system drive. The CachyOS system drive has: a **4GB vfat** ESP partition (nvme0n1p1), a **~950GB btrfs** partition (nvme0n1p2), and Btrfs subvolumes including `@`, `@home`, `@cache`, `@root`. Data/gaming drives are typically NTFS or have different partition sizing. Don't assume `/dev/nvme1n1p2` is root — check MODEL and partition sizes.
- **Pacman cache cleanup from live environment needs `@cache` subvol** — In the live overlay, `/var/cache/pacman/pkg/` is ephemeral. The real cache is on the `@cache` Btrfs subvol. To clean it: `mount -t btrfs -o subvol=@cache /dev/nvme0n1p2 /mnt/root/var/cache && paccache -rvk 0 -c /mnt/root/var/cache/pacman/pkg/`. Without mounting the correct subvol, you'll clean the live overlay (which is empty) and think you've freed space you didn't.
- **`limine-snapper-sync` fails when `/.snapshots` is not a Btrfs subvolume** — The `snapper-cleanup.service` ExecStopPost hook runs `limine-snapper-sync --no-force-save` which calls `btrfs subvolume show /.snapshots`. If `/.snapshots` is a regular directory (or the root itself, inode 2) instead of a Btrfs subvolume, this fails with exit code 1 and triggers a popup. The snapper cleanup itself succeeds — only the limine sync hook fails. Fix: disable the hook via systemd drop-in (see `references/cachyos-snapper-limine-sync-failure.md` in system-administration skill).
- **`limine-entry-tool` fails in chroot without `/dev` bind-mounted** — The Limine entry regeneration tool needs `/dev/fd/63` and other device files. In a chroot from a live environment: `mount --bind /dev /mnt/root/dev` before `chroot /mnt/root limine-entry-tool`. Without this, it errors with `Fatal error: Failed to create the main Isolate. (code 32)` and `No such file or directory` for `/dev/fd/63`. If the tool keeps failing, fall back to direct `sed` on `/boot/limine.conf` to fix `default_entry` or remove stale snapshot entries. The bootloader config format is Limine-native (not GRUB), and sed edits are safe as long as you know the entry structure.
- **Snapper restore fails with `10-limine-reset-enroll` hook error** — The pre-hook checks for ESP at `/boot` and `/boot/limine.conf`. If the ESP is mounted elsewhere (`/boot/efi`) or not mounted, the hook fails with exit code 1 and blocks the restore. Fix: mount the ESP at `/boot` (use bind mount if already at `/boot/efi`), or temporarily `chmod -x` the hook. The hook belongs to the `cachyos-sbctl` package and resets secure boot enrollment during snapshot rollback — it's safe to skip temporarily, but re-enable afterward.
  - **Common hidden cause: fstab `/boot` entry is commented out** — The CachyOS installer (Calamares) can leave the `/boot` fstab entry commented (`#UUID=... /boot vfat ...`). Even with the ESP correctly formatted, it never mounts on boot because the entry is silenced. When diagnosing this error in a live/rescue environment, always check the fstab first: `grep -E "(boot|vfat)" /mnt/root/etc/fstab` — if the entry starts with `#`, uncomment it. This is the root cause of "worked before, broken after reboot" scenarios.
  - **`ESP_PATH` is forced to `/boot` via `/etc/default/limine`** — The hook's `load_config()` function reads `/etc/default/limine` which explicitly sets `ESP_PATH="/boot"`, overriding any auto-detection via `bootctl --print-esp-path`. So even when the ESP is correctly mounted elsewhere (e.g. `/boot/efi`), the hook checks `/boot` specifically. Fix: `mount` the ESP at `/boot`, not `/boot/efi`, or bind-mount.

## SSH Verbose Diagnostic Pattern

When `ssh -v` hangs at "Authenticating to...":

```
debug1: Connection established.
debug1: identity file /home/rurouni/.ssh/id_ed25519 type 3
debug1: Authenticating to 192.168.1.111:22 as 'raymond'
[then hangs → timeout]
```

This means TCP works, version negotiation passes, key is loaded locally — but the server's authorized_keys for that user doesn't contain your public key. Three causes:
1. Key not in authorized_keys (most common)
2. authorized_keys file or its parent `.ssh/` directory has wrong permissions (`.ssh` must be 700, `authorized_keys` must be 600)
3. `AuthorizedKeysFile` in sshd_config points to a non-standard path (rare on Arch)

**The `-vvv` escalation pattern:** Use vanilla first → `-v` if timeout → `-vvv` if port confirmed open but still hangs. The exact line where verbose output stops tells you which phase is failing.