# Photon Upstream Degraded — Diagnosis & Recovery

## Symptoms

- First iMessage through Photon works, subsequent messages don't
- `hermes photon status` shows all green (token ✓, sidecar deps ✓, project ✓)
- Gateway is `active` (systemd), sidecar Node process is running
- No Photon-related errors in `gateway.log` (if `logging.level: WARNING`, INFO messages like `"[photon] connected"` are suppressed)

## Diagnostic Signal

The adapter has a ~15-minute degradation window before the upstream drops:

```
ERROR hermes_plugins.photon_platform.adapter: [photon] Photon upstream stream degraded
  (state=degraded, degradedForMs=13692):
  [spectrum.stream] stream persistently failing; still retrying
  ConnectionError: [upstream] upstream connect error or disconnect/reset before headers.
  reset reason: remote connection failure, transport failure reason:
  delayed connect error: Connection refused
```

Then the adapter loops retrying forever — sidecar alive, pipe dead.

## Root Cause

The upstream gRPC stream to Photon's Spectrum Cloud (`spectrum.photon.codes`) drops, and the sidecar's `spectrum-ts` SDK can't recover. Exact trigger is unclear — could be:
- Photon Spectrum session credential expiry
- Transient Photon-side outage that the SDK's retry logic doesn't survive
- Sidecar crash (code -9, SIGKILL) that orphans the gRPC stream

## Recovery

### 1. Confirm the state

```bash
# Check gateway logs for Photon errors (journal captures all levels)
journalctl --user -u hermes-gateway --since "1h" | grep -i photon

# Check if sidecar is running
ps aux | grep 'sidecar\|index\.mjs' | grep -v grep
```

### 2. Workaround: Force-restart the gateway

`systemctl --user restart hermes-gateway` is **blocked** from inside the gateway's terminal sandbox. Use `execute_code` (Python subprocess) to send SIGTERM, then systemd auto-restarts:

```python
import os, signal, time
# Find the gateway PID first
gw_pid = 3230248  # from ps aux | grep 'gateway run'
os.kill(gw_pid, signal.SIGTERM)
time.sleep(8)
```

If SIGTERM doesn't exit within ~20s (gateway can get stuck in deactivating state), send SIGKILL in a second call:
```python
import os, signal
gw_pid = 3230248
os.kill(gw_pid, signal.SIGKILL)
```

### 3. Verify recovery

```bash
# Wait for systemd restart (~5-10s)
sleep 5
systemctl --user is-active hermes-gateway  # should be "active"
ps aux | grep 'sidecar' | grep -v grep     # new sidecar PID
journalctl --user -u hermes-gateway --since "30 seconds ago" | grep -i photon
```

### 4. Confirm adapter connected

The INFO-level `"[photon] connected — sidecar on 127.0.0.1:8789"` message is suppressed when `logging.level: WARNING`. Workarounds:
- Temporarily set `logging.level: INFO` in `config.yaml` and restart
- Or rely on absence of ERROR messages + sidecar process running as evidence

## Log-Level Masking Pitfall

```yaml
logging:
  level: WARNING   # suppresses ALL INFO messages including photon "connected" line
```

This makes it look like Photon never initialized when it actually did. Check for lack of ERRORs instead, or grep the full log for any Photon line to confirm the adapter is running.

## Prevention

If the degraded state recurs consistently at ~15-minute intervals, the root cause is likely a Photon Spectrum session expiry — contact Photon support. The `logging.level: WARNING` config should be lowered to `INFO` temporarily during diagnosis to capture the full connection lifecycle.