(Gj0(dZddlmZddlZddlZddlZddlmZmZddlm Z ddl m Z m Z ddl mZddlmZejeZd Zejd d#dZd$d%dZddddd&dZddd'dZd(dZddd'dZddd'd Zddd)d"ZdS)*u External drain-control marker contract (dashboard → gateway). Task 2.2 of the safe-shutdown plan (decisions.md Q-B, option A): the dashboard has no way to call into a running gateway — there is no HTTP control channel into the gateway process (guardrails: "there is NO external control channel into a running gateway"). Restart/drain is driven only by the gateway reacting to its own inputs: slash commands, process signals, and file markers it writes itself (``.restart_notify.json``). So the begin/cancel-drain dashboard endpoint communicates with the running gateway the same way: it writes (or removes) a marker file, and a gateway background watcher reacts to it. This module owns that marker contract so both sides — the dashboard endpoint (writer) and the gateway watcher (reader) — share one definition and can never disagree. Contract (presence-based, mirroring ``.restart_notify.json``): * begin-drain → write ``{HERMES_HOME}/.drain_request.json`` with ``{"action": "drain", "requested_at": , "principal": , "epoch": , "suppress_notification": }``. * cancel-drain → remove the marker. * The gateway watcher treats **presence of a marker stamped with the current instantiation epoch** as "external drain active": flip ``gateway_state -> "draining"`` and stop accepting new turns. Absence (or a marker from a *prior* instantiation) means "not draining" (revert to ``running`` if we had flipped it). Why the epoch (NS-570). ``HERMES_HOME`` is a **durable** store — on Hermes Cloud it is a persistent Fly volume (``/opt/data``). A begin-drain marker written there *survives a machine restart*. But the disruptive lifecycle actions a drain protects (auto-update / image migrate / env edit / profile change) all **restart the machine**, which is exactly the signal that the drain is over. Without the epoch, a freshly-restarted gateway re-reads the orphaned marker on boot and parks itself right back in ``draining`` forever (NS-570: an auto-updated instance refused every turn for ~52 min). Stamping the marker with an identity of *this* container/VM instantiation, and ignoring a marker whose epoch doesn't match, makes "a deliberate restart clears the drain" true by construction — while a marker written during the *current* instantiation (the live drain) still matches, and an s6 respawn of just the gateway (PID 1 / init unchanged) still honours an in-flight drain. Reading the marker never raises: a malformed/half-written file reads as "present but contentless", which the watcher still treats as drain-active (fail-safe toward quiescing — a corrupt begin marker must not be ignored). The epoch check is deliberately **lenient**: it ignores a marker only on a *definite* epoch mismatch. A marker with no epoch (legacy/corrupt/contentless), or an environment where the epoch cannot be computed (non-Linux, no ``/proc``), both degrade to the original presence-only behaviour — never fail-closed. ) annotationsN)datetimetimezone)Path)AnyOptional)get_hermes_home)atomic_json_writez.drain_request.json)maxsizereturnstrcd} tdd}n#t$rYnwxYwd} tdd}|ddd}|d}n#tt f$rYnwxYw|s|sdS|d |S) u!Identity of THIS container / VM instantiation. Stable for the life of the PID-1 init process — so an s6 respawn of just the gateway keeps the same epoch and an in-flight drain is honoured — but changes when the machine/container is recreated (a fresh PID 1 → a fresh epoch). Composed from two ``/proc`` facts: * the kernel **boot id** (``/proc/sys/kernel/random/boot_id``) — changes on a VM / microVM reboot (e.g. a Fly Firecracker machine restart); * **PID 1's start time** (field 22 of ``/proc/1/stat``) — changes on a plain ``docker restart`` (the host kernel, hence boot_id, is unchanged, but ``/init`` is a brand-new process). Together they discriminate every restart mode that matters: | event | boot_id | pid1 start | epoch | marker | |--------------------------------|---------|------------|--------|--------| | Fly microVM reboot (auto-upd.) | changes | changes | NEW | reject | | plain ``docker restart`` | same | changes | NEW | reject | | s6 respawn of the gateway only | same | same | SAME | honour | | host ``hermes gateway restart``| same | same(init) | SAME | honour | The last row is intentional: a host install has no durable-volume drain bug, and honouring a drain across a deliberate process restart is the intended reversible behaviour (D4a) — PID 1 there is the long-lived init (systemd/launchd), so the epoch is stable. Returns ``""`` when neither identity source is readable (non-Linux, no ``/proc``). An empty epoch disables the staleness check downstream, degrading to the released presence-only behaviour — never fail-closed. Memoised: the epoch is constant for the life of the process. z/proc/sys/kernel/random/boot_idutf-8encodingz /proc/1/stat)r :)r read_textstripOSErrorrsplitsplit IndexError)boot_id pid1_startstattails ;/home/rurouni/.hermes/hermes-agent/gateway/drain_control.pycurrent_instantiation_epochr"CsDG 2 3 3 YY ( ( UWW       J  N##--w-??{{3""1%++--"X Z       :r $ $ $ $$s"5: AA AB''B;:B;homeOptional[Path]rcT||n t}t|tz S)zBAbsolute path to the drain-request marker, respecting HERMES_HOME.)r r_DRAIN_REQUEST_FILENAME)r#bases r!drain_request_pathr(s(#44):):D ::/ //z drain-controlF) principalsuppress_notificationr#r*r+booldict[str, Any]cdtjtj|t t |d}tt|||S)u Write the begin-drain marker. Returns the payload written. Atomic write so the gateway watcher never reads a half-written file. Idempotent: re-writing while a drain is already in progress just refreshes ``requested_at`` (harmless — the watcher keys off presence, not content). Stamps the marker with :func:`current_instantiation_epoch` so a marker that later survives a machine restart on the durable HERMES_HOME volume can be recognised as stale and ignored (NS-570). ``suppress_notification`` is a generic "be quiet on the shutdown that ends this drain" flag. When the drain culminates in a process exit (e.g. NAS recreates the machine for an auto-update image migration), the gateway's shutdown path reads it via :func:`drain_notification_suppressed` and skips the *home-channel* "gateway shutting down" broadcast — the operator-flavoured ping that would otherwise fire on every routine auto-update, potentially dozens of times a day. It NEVER suppresses the per-active-session interrupt ping. The gateway stays agnostic about *why* the drain is quiet; the policy of which drain causes set the flag lives entirely in the caller (NAS). The field defaults False so legacy/operator drains behave exactly as before. drain)action requested_atr*epochr+) rnowrutc isoformatr"r,r r()r*r+r#payloads r!write_drain_requestr7sf8 X\22<<>>,..!%&;!rawr?datas r!rHrHs d # #Dnngn.. tt  ;T1EEEtttttz#  " dD)) 144r1s+( A% A%A  A%)A>>BB)r r)N)r#r$r r)r*rr+r,r#r$r r-)r#r$r r,)rAr-r r,)r#r$r rM)__doc__ __future__r functoolsrOloggingrrpathlibrtypingrrhermes_constantsr utilsr getLogger__name__r<r& lru_cacher"r(r7r@rFrJrLrHr)r!rcs00b#""""" '''''''' ,,,,,,######w""/Q:%:%:% :%z00000%"' ######L37       ####*/3&=A33333322622222222r)