# Telegram Polling Conflict — Permanent Fix

## Full Diagnostic Pattern

### Indicators

| Signal | Value |
|---|---|
| `getMe` | `ok:true` — token is valid |
| Network | api.telegram.org reachable (0% loss, ~168ms RTT) |
| Gateway state | Systemd `failed` — exits with code 1 immediately or within ~2 min |
| Journal logs | `Telegram polling conflict (1/5) — previous session still held open` |
| Journal logs (process exit) | `Shutdown context: signal=SIGTERM under_systemd=yes` then `exited, status=1/FAILURE` |
| Systemd restart loop | RestartSec=5, no rate limiting (StartLimitIntervalSec=0), cycles forever |

### How Telegram Long-Poll Conflicts Work

1. The gateway calls `getUpdates` (long-poll, timeout ~50s) — Telegram's server keeps the connection open waiting for new messages
2. When the gateway process is killed (crash, SIGTERM, OOM), Telegram's server holds the session open for ~30s before noticing the connection dropped
3. A new gateway process starts immediately and calls `getUpdates` again — Telegram rejects it with `409 Conflict: terminated by other getUpdates request`
4. PTB's `_handle_polling_conflict` enters a retry loop: 5 attempts, delays of 15s → 25s → 35s → 45s → 55s (175s total)
5. Systemd sends SIGTERM before retries complete → cycle repeats

### Why Existing Fixes Fail

- **"Double restart with pause"** — vulnerable to timing; if both restarts happen within Telegram's 30s stale session window, both fail
- **`drop_pending_updates=True`** — PTB uses this on initial start, but it only affects the *polling transition*, not the stale *server-side session*
- **Sheer retry count** — 5 retries over 175s is too slow; systemd's expectations and user patience both expire

## Two-Part Fix

### Part 1: Immediate — Clear Telegram's Stale Session

```bash
TOKEN=$(grep -m1 '^TELEGRAM_BOT_TOKEN=' ~/.hermes/.env | cut -d= -f2- | tr -d '"')

# Delete webhook + drop pending updates
curl -s "https://api.telegram.org/bot${TOKEN}/deleteWebhook?drop_pending_updates=true"

# Acknowledge all pending updates — releases the stale long-poll session
curl -s "https://api.telegram.org/bot${TOKEN}/getUpdates?offset=-1&timeout=1"
```

After this, restart the gateway: `systemctl --user restart hermes-gateway`.

**Do NOT call `getUpdates` during diagnosis** — it creates a competing polling session that causes the exact conflict you're trying to fix. Always use `getMe` for token tests and `sendMessage` for outbound tests.

### Part 2: Permanent — Systemd ExecStartPre

Create `~/.hermes/scripts/telegram-clear-session.sh`:

```bash
#!/bin/bash
# Clear stale Telegram polling session before gateway starts.
# Called as ExecStartPre in hermes-gateway.service.

ENV_FILE="${HERMES_HOME:-$HOME/.hermes}/.env"
if [ ! -f "$ENV_FILE" ]; then
    echo "telegram-clear-session: .env not found at $ENV_FILE"
    exit 0
fi

TOKEN=$(
    sed -n 's/^TELEGRAM_BOT_TOKEN=//p' "$ENV_FILE" | head -1 | tr -d "'\""
)
if [ -z "$TOKEN" ]; then
    echo "telegram-clear-session: TELEGRAM_BOT_TOKEN not found in $ENV_FILE"
    exit 0
fi

API="https://api.telegram.org/bot${TOKEN}"

# 1. Delete webhook + drop pending updates
echo "telegram-clear-session: Calling deleteWebhook..."
curl -s --connect-timeout 5 --max-time 10 \
    "${API}/deleteWebhook?drop_pending_updates=true" -o /dev/null -w "  HTTP %{http_code}\n"

# 2. Acknowledge all pending updates via getUpdates with a high offset
echo "telegram-clear-session: Calling getUpdates with high offset..."
curl -s --connect-timeout 5 --max-time 10 \
    "${API}/getUpdates?offset=-1&timeout=1" -o /dev/null -w "  HTTP %{http_code}\n"

echo "telegram-clear-session: Done — Telegram stale session cleaned."
exit 0
```

Modify `~/.config/systemd/user/hermes-gateway.service`:

```ini
[Service]
Type=simple
ExecStartPre=/home/rurouni/.hermes/scripts/telegram-clear-session.sh
ExecStart=/home/rurouni/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run
```

Then:

```bash
chmod +x ~/.hermes/scripts/telegram-clear-session.sh
systemctl --user daemon-reload
systemctl --user restart hermes-gateway
```

### Verification

```bash
# Check the pre-start ran
journalctl --user -u hermes-gateway --since "2m ago" | grep "telegram-clear-session"
# Expected: HTTP 200 for both calls

# Wait 15s, check no conflicts
journalctl --user -u hermes-gateway --since "30s ago" | grep -i "polling conflict"
# Expected: empty (no conflicts)

# Confirm gateway stays active
systemctl --user is-active hermes-gateway
# Expected: active (running for 15+ seconds)
```

## Why It Works

- `deleteWebhook?drop_pending_updates=true` — tells Telegram's server to discard the old long-poll session and clear any queued updates
- `getUpdates?offset=-1` — acknowledges the last update ID, telling Telegram's server "I've seen everything up to now" — this effectively terminates the stale session from Telegram's perspective
- Running this **before** the gateway starts means the PTB library's `start_polling()` call is the first and only getUpdates session — no conflict possible
- The script is idempotent and fast (~0.5s total) — adds no noticeable delay to gateway startup

## Edge Cases

- **If the `.env` file doesn't exist or has no token:** script exits 0 — the gateway will start and fail normally with "no token configured"
- **If Telegram's API is unreachable:** script exits 0 — the gateway will start and hit a network error, which is handled normally by PTB's reconnect ladder
- **If multiple restarts happen in rapid succession:** each one clears the previous session before starting — no conflict accumulation