# Hermes Diagnostics — Telegram Diagnosis

## Gateway Service Health

**DO NOT check config first.** Always check the *process*, not the *configuration*.

```bash
systemctl --user is-active hermes-gateway
systemctl --user status hermes-gateway --no-pager | tail -15
```

**If active:** proceed. **If inactive:** `systemctl --user restart hermes-gateway`. **If failed:**
```bash
systemctl --user reset-failed hermes-gateway
systemctl --user restart hermes-gateway
```

After restart, verify Telegram is actually connected:
```bash
sleep 5
GATEWAY_PID=$(pgrep -f 'hermes.*gateway.*run' | head -1)
ss -tnp | grep "${GATEWAY_PID}" | grep "149.154.166"
```
Expected: 2+ ESTABLISHED connections to `149.154.166.110:443`.

## Bot Token Validity

```bash
source <(grep -a '^TELEGRAM_BOT_TOKEN' ~/.hermes/.env)
curl -s --max-time 10 "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/getMe"
```
Expected: `{"ok":true,"result":{"id":...,"is_bot":true,...}}`

**404:** token malformed/clipped, bot deleted, or token revoked. **401:** token revoked — regenerate via BotFather.

### Distinguishing 404 Causes

| getMe Response | Meaning |
|---|---|
| `ok:true, result:{id...,is_bot:true}` | Token valid, bot working |
| `ok:false, error_code:401` | Token revoked by BotFather |
| `ok:false, error_code:404` | Bot doesn't exist on Telegram's servers — deleted, token clipped, or never created |
| `ConnectError / Timeout` | Network issue — can't reach api.telegram.org |

A 404 alone doesn't tell you *why* the token is bad. Combine with:
- **Timeline reconstruction** (see below) — if `.env` was modified after the bot last worked, the change may have corrupted it
- **Pre-update backup comparison** — check if a pre-update snapshot has a different token
- **`stat ~/.hermes/.env`** — compare `.env` birth/modify time to last known-good Telegram activity

### `***` in config.yaml — distinguish masking from real corruption

When `~/.hermes/config.yaml` shows `gateway.telegram.token: 8796348132:***`, the `***` may be Hermes terminal-output masking OR literal file contents. Verify with raw hex:

```bash
sed -n '<LINE>' ~/.hermes/config.yaml | od -c
# If output shows `*   *   *` → literal asterisks in the file = corrupted token
# If output shows real alphanumeric chars → display masking only, token is fine

# Also check .env:
grep 'TELEGRAM_BOT_TOKEN' ~/.hermes/.env
```

If `***` is literal (hex `2a2a2a`), the token secret has been replaced — the original is unrecoverable. Regenerate via BotFather.

### Token Format Validation (Under Redaction)

Hermes redacts tokens in both `read_file` and `terminal` output. To validate format when you can't see the full value:

```bash
python3 -c "
e = open('/home/rurouni/.hermes/.env').read()
import re
m = re.search(r'TELEGRAM_BOT_TOKEN=(.*)', e)
if m:
    t = m.group(1)
    print(f'Token length: {len(t)}')
    print(f'Has colon: {\":\" in t}')
    print(f'Format regex: {bool(re.match(r\"\d+:[A-Za-z0-9_-]{30,}$\", t))}')
"
```

A valid token is `\d+:[A-Za-z0-9_-]{30,}` — ~46 chars total. If format validates but API returns 404, the bot was deleted/revoked on Telegram's side.

## Timeline Reconstruction

When the token returns 404 but *was working before*, reconstruct the timeline to understand what changed:

```bash
# 1. When was .env last modified?
stat ~/.hermes/.env | grep -E 'Modify|Birth'

# 2. When was the last successful Telegram activity?
journalctl --user -u hermes-gateway --no-pager | grep -E 'Telegram' | tail -5

# 3. Any recent Hermes updates?
ls -la ~/.hermes/state-snapshots/ 2>/dev/null | tail -5

# 4. Was the token different before the update?
PRE=$(ls -t ~/.hermes/state-snapshots/ | head -1)
if [ -n "$PRE" ]; then
  diff <(grep TELEGRAM_BOT_TOKEN ~/.hermes/state-snapshots/"$PRE"/.env 2>/dev/null) \
       <(grep TELEGRAM_BOT_TOKEN ~/.hermes/.env 2>/dev/null)
fi
```

Key patterns from real cases:
- `.env` created **after** Telegram last worked → token was in `config.yaml` (now corrupted or overwritten)
- `.env` modified same day Telegram stopped → likely a manual token change or config edit
- `.env` never modified, no pre-update backup → token revoked on Telegram's side (BotFather or bot deletion)

## Webhook Conflict

Long-poll and webhook are mutually exclusive.
```bash
source <(grep -a '^TELEGRAM_BOT_TOKEN' ~/.hermes/.env)
curl -s "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/getWebhookInfo"
```
Expected: `"url":""`. If set, clear: `curl -s "https://api.telegram.org/bot${TOKEN}/deleteWebhook"`

## Active Connections to Telegram

```bash
GATEWAY_PID=$(pgrep -f "hermes.*gateway.*run" | head -1)
ss -tnp | grep "${GATEWAY_PID}" | grep "149.154.166"
ls -la /proc/${GATEWAY_PID}/fd/ | grep "149.154"
```
Expected: 2-3 ESTABLISHED connections.

## Log File Health

```bash
ls -la ~/.hermes/logs/gateway.log
grep -a "telegram\|send\|failed\|error\|warning" ~/.hermes/logs/gateway.log | tail -20
```
Stale mtime with live process = normal when logging.level is WARNING+.

### Reading Gateway Logs When They're Binary

```bash
# Check if binary file
file ~/.hermes/logs/gateway.log  # → "ASCII text" vs "data"
# grep with -a for binary files
grep -a "telegram" ~/.hermes/logs/gateway.log | tail -20
# Agent.log is often binary due to streaming/ANSI content
grep -a "telegram\|flood\|network\|getMe\|404\|401" ~/.hermes/logs/agent.log | tail -20
# Errors.log is usually clean text
grep -i "telegram" ~/.hermes/logs/errors.log | tail -20
```

### Zero Telegram Log Entries in Running Gateway

If gateway is `active` (systemd) but has **zero** Telegram-related log entries across all log files:

- **Token failed authentication at init.** The Telegram adapter initializes, calls `getMe`, gets 404/401, and silently falls into reconnection without logging an explicit "auth failed" message.
- This is distinct from "gateway crashed" (systemd `failed` state) and "network error" (logged connect attempts + fallback IP retries).
- **Fix:** Fix the token. Process restart alone won't help if the token is still invalid.

Distinguish from network failure:

| Pattern | Log Evidence |
|---|---|
| Token auth failure | Zero Telegram entries anywhere; gateway process alive |
| Network failure | `Primary api.telegram.org connection failed; trying fallback IPs` |
| Flood control | `Telegram flood control, waiting Xs` or `flood control on send` |

## Outbound Send Test

```bash
source <(grep -a '^TELEGRAM_BOT_TOKEN' ~/.hermes/.env)
curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
  -d "chat_id=<USER_CHAT_ID>&text=Diagnostic test from Hermes"
```
Expected: `{"ok":true,...}`.

**Known failures:** 413 (too large), 403 (bot blocked by user), 429 (flood control).

## Inbound Message Test

After outbound confirmed, have user send a message, then:
```bash
source <(grep -a '^TELEGRAM_BOT_TOKEN' ~/.hermes/.env)
curl -s "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/getUpdates?offset=-1&timeout=2"
```
Empty result with running gateway is normal (gateway consumed the update).

## Session Store Check

```bash
python3 -c "
import sqlite3
conn = sqlite3.connect('/home/rurouni/.hermes/state.db')
c = conn.cursor()
rows = c.execute(\"SELECT id, session_id, platform, created_at FROM sessions WHERE platform='telegram' ORDER BY created_at DESC LIMIT 3\").fetchall()
for r in rows: print(r)
conn.close()
"
```

## api_server Env Var vs Config Mismatch

The API server reads its key from `API_SERVER_KEY` env var, NOT from `config.yaml`'s `api_server.key`. Fix:
```bash
API_KEY=$(grep -A2 'api_server:' ~/.hermes/config.yaml | grep 'key:' | awk '{print $2}')
sed -i "/^API_SERVER_KEY=/d" ~/.hermes/.env
echo "API_SERVER_KEY=${API_KEY}" >> ~/.hermes/.env
```
Then `/restart` the gateway.

## Silent Polling Wedge Detection

**Symptoms:** Gateway process alive, health returns 200, but log entries stopped hours ago. No inbound messages.

**Detection:**
```bash
# 1) Confirm gateway is alive
systemctl --user is-active hermes-gateway
curl -s http://localhost:18789/health

# 2) Check when Telegram last logged
ls -la ~/.hermes/logs/gateway.log

# 3) Look for fatal error signature
grep -a "could not reconnect after\|telegram_network_error\|retryable-fatal" ~/.hermes/logs/gateway.log | tail -3

# 4) Flood control in run-up
grep -a "flood control" ~/.hermes/logs/gateway.log | tail -5

# 5) Verify bot API works
source <(grep -a '^TELEGRAM_BOT_TOKEN' ~/.hermes/.env)
curl -s --max-time 10 "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/getMe"
```

**Fix:** `systemctl --user restart hermes-gateway`

**Verify:**
```bash
sleep 15
systemctl --user is-active hermes-gateway
```

**If still no Telegram log entries within 60s:** check `agent.log`:
```bash
grep -a "telegram" ~/.hermes/logs/agent.log | tail -5
hermes send --platform telegram --target "Raymond" --message "🟢 Test from restarted gateway"
```

## Polling Conflict (409) During Restart

```bash
grep -a "polling conflict" ~/.hermes/logs/agent.log | tail -3
```

**Fix — double restart with pause:**
```bash
systemctl --user restart hermes-gateway
sleep 30
systemctl --user restart hermes-gateway
sleep 25
```

**Critical rule:** Never run manual `getUpdates?offset=-1&timeout=N` curl during diagnosis — creates competing long-poll sessions. Use `sendMessage` for tests.

## Key Pitfalls

- `failed` state: `systemctl --user restart` alone may not work — run `reset-failed` first
- Configured token ≠ working bot — check the **process**, not the **config**
- `***` in config can be literal file corruption, not display masking — verify with `od -c`
- Zero Telegram log entries in a running gateway = authentication failed at initialization
- After restart, new process writes to `agent.log`, NOT `gateway.log`
- Reconnect ladder: 10 attempts (5s→60s backoff). After exhausting all 10, `_set_fatal_error` fires but systemd does NOT auto-restart
- Flood control is tracked separately from network errors; can accumulate over hours
- Manual `getUpdates` calls break the gateway's polling (409 Conflict)
- `.env` birth time can tell you if the token was ever in `.env`: `stat ~/.hermes/.env | grep Birth`
- **Token format validates but API returns 404** = bot was deleted/revoked on Telegram's side. Regenerate via BotFather. This is a different root cause from `***` corruption, network failure, or polling wedge.
- **journalctl may show no entries** immediately after a gateway restart if the restart was too recent — check `since "5 minutes ago"` or use `--no-pager` with broader time range