# Provider Failure Diagnosis: "Model provider failed after retries"

## Symptom

User sees `⚠️ The model provider failed after retries.` — the gateway warns but doesn't surface the raw error detail.

## Three-Layer Log Architecture

Hermes has three distinct log sources. Provider API failures may appear in **any** of them:

| Layer | Location | What it captures |
|-------|----------|-----------------|
| Gateway flat log | `~/.hermes/logs/gateway.log` | Platform-level events (Telegram connect/disconnect, API server auth, hygiene compression) |
| Agent flat log | `~/.hermes/logs/agent.log` | Session-level events (tool errors, model switches, auxiliary client calls) |
| Gateway journal (systemd) | `journalctl _PID=<pid>` | **Actual provider API call failures** — 401s, 429s, 404s, stream drops, retry loops |

**Critical insight:** The gateway.log is a **platform log**, not a provider log. It records whether Telegram connected, not whether the DeepSeek API call succeeded. The provider call failures live in the gateway process's stdout/stderr, which systemd captures into the journal.

## Diagnostic Procedure

### 1. Fast check — which model/provider is this session using?

Look at the current session's system prompt — `model:` line near the top.

### 2. Check gateway journal for API call failures

```bash
# Find the gateway PID
ps aux | grep 'gateway run' | grep -v grep

# Tail its journal for provider errors
journalctl _PID=<pid> --no-pager -n 100 | grep -iE '401|429|403|500|502|503|timeout|retry|auth|fail|error'
```

Key patterns to grep for:
- `AuthenticationError` — API key is wrong/expired (HTTP 401)
- `RateLimitError` — quota exhausted or rate limited (HTTP 429)
- `NotFoundError` — model not found at endpoint (HTTP 404)
- `Empty response` — model returned nothing (local models, streaming issues)
- `peer closed connection` — upstream dropped the connection mid-stream (CloudFront/gateway timeout)

### 3. Verify the API key

```bash
# Check if env var is set in the gateway's environment
cat /proc/<pid>/environ 2>/dev/null | tr '\0' '\n' | grep DEEPSEEK

# Check the .env file (loaded by Hermes at gateway startup)
grep DEEPSEEK_API_KEY ~/.hermes/.env

# Test the key directly
curl -s -w '\nHTTP:%{http_code}' https://api.deepseek.com/v1/models \
  -H "Authorization: Bearer $(grep DEEPSEEK_API_KEY ~/.hermes/.env | cut -d= -f2-)"
```

### 4. Check agent.log for cross-reference

```bash
# Look for the same provider+model string
grep -a 'deepseek-v4-flash\|provider=deepseek' ~/.hermes/logs/agent.log | tail -20
```

### 5. Check gateway.log for connectivity issues

```bash
tail -100 ~/.hermes/logs/gateway.log | grep -iE 'telegram.*fail|reconnect|dns|timeout|blocked'
```

Sometimes the provider is fine but Telegram connectivity itself is the bottleneck — a stale gateway.log mtime on disk while the process is alive can mean the platform is wedged, not the provider.

## Common Failure Signatures

| Error | Meaning | Fix |
|-------|---------|-----|
| `HTTP 401: Authentication Fails, Your api key: ****red. is invalid` | Key expired/revoked | Regenerate key in DeepSeek dashboard, update `~/.hermes/.env` |
| `HTTP 401: Invalid API key` | Key wrong for this endpoint | Check provider base_url vs key origin |
| `HTTP 429: Monthly usage limit reached` | Free tier exhausted | Top up or switch model |
| `HTTP 404: no router for requested model` | Model name doesn't match llama-swap router | Update config.yaml model ID or add router entry |
| `Empty response (no content or reasoning) — retry N/3` | Local model returned nothing | Check if the model is loaded (llama-swap status) |
| `peer closed connection without sending complete message body` | Upstream timeout / CloudFront cut | Retry; increase max_tokens or reduce context |
| `Flow control exceeded. Retry in N seconds` | Telegram API rate limit | Not a provider issue — back off on message frequency |

## Pitfalls

- **gateway.log does NOT contain provider API call results** for remote providers. Don't waste time grepping for HTTP status codes there. Provider failures only appear in the journal or agent.log.
- **agent.log is a binary file** — use `grep -a` not plain `grep`.
- **`DEEPSEEK_API_KEY` length may show as 0** in a CLI agent session even when the gateway has it properly loaded. The CLI agent doesn't inherit the gateway's process environment. Test via `/proc/<gateway_pid>/environ` instead.
- **Gateway restart changes PID** — journalctl by PID only works for the current running process. After restart, use `journalctl --user -u hermes-gateway --since "5 min ago"`.
- **Key rotations** — DeepSeek keys occasionally expire. If you see new 401s on a previously working setup, check the key first.