Jjy0UdZddlZddlZddlZddlZddlZddlZddlZddlZddl m Z m Z ddl m Z ddlmZmZejdkrddlmZnddl mZdd lmZmZd aejZd Zd Zd ZdZdedzde fdZ!dZ"de#ddfdZ$dDdZ%dDdZ&e&Gddej'Z(dddddddZ)dddddd d d!ee d"ee#d#ee*d$ee*d%ee#d&e de fd'Z+dDd(Z,Gd)d*eZ-da.d+e/d,<da0ee e/d-<gZ1e2e/d.<d a3ej4Z5Gd/d0e Z6dDd1Z7dDd2Z8d3ej9ddfd4Z:dDd5Z;dEd7edDd:Z?dd;de*d?e*d$e*d@ejAdAeej'ddfdBZBdCZCdS)Fu Centralized logging setup for Hermes Agent. Provides a single ``setup_logging()`` entry point that both the CLI and gateway call early in their startup path. All log files live under ``~/.hermes/logs/`` (profile-aware via ``get_hermes_home()``). Log files produced: agent.log — INFO+, all agent/tool/session activity (the main log) errors.log — WARNING+, errors and warnings only (quick triage) gateway.log — INFO+, gateway-only events (created when mode="gateway") gui.log — INFO+, dashboard/websocket/TUI-gateway events (created when mode="gui") All files use ``RotatingFileHandler`` with ``RedactingFormatter`` so secrets are never written to disk. Component separation: gateway.log only receives records from ``gateway.*`` loggers — platform adapters, session management, slash commands, delivery. gui.log receives dashboard-side records from ``hermes_cli.web_server``, ``hermes_cli.pty_bridge``, ``tui_gateway.*``, and ``uvicorn.*``. agent.log remains the catch-all (everything goes there). Session context: Call ``set_session_context(session_id)`` at the start of a conversation and ``clear_session_context()`` when done. All log lines emitted on that thread will include ``[session_id]`` for filtering/correlation. N) QueueHandler QueueListener)Path)OptionalSequencewin32)ConcurrentRotatingFileHandler)RotatingFileHandler)get_config_pathget_hermes_homeFz>%(asctime)s %(levelname)s%(session_tag)s %(name)s: %(message)szC%(asctime)s - %(name)s - %(levelname)s%(session_tag)s - %(message)sc*tj}t|ddpd}|dddvr|S t|dd}|"t j|ddd }d |_|Sn#t$rYnwxYw|S) uReturn a stderr stream that tolerates Unicode on all platforms. On Windows the console encoding is often a legacy MBCS codec (cp949, cp1252, …) that raises ``UnicodeEncodeError`` for characters like the em-dash (U+2014). We wrap ``sys.stderr`` in a ``TextIOWrapper`` with ``errors='replace'`` so log lines are never lost — un-encodable characters are replaced with ``?`` instead of crashing the process. encodingNutf-8-)utf8utf8surrogateescapebufferreplaceT)rerrorsline_bufferingcdSNr4/home/rurouni/.hermes/hermes-agent/hermes_logging.pyz_safe_stderr..ssDr) sysstderrgetattrlowerrio TextIOWrapperclose Exception)streamrbufwrappeds r _safe_stderrr)YsZFvz400;GH~~R((,KKK  fh-- ?&  # G)LGMN        Ms 4B BBz%Cannot acquire lock after 20 attemptsexcreturncxtjdko*t|tott |vS)aReturn True for concurrent-log-handler's Windows lock timeout. On Windows Desktop, slash-command workers and the gateway can all write to the same rotating log files. ``concurrent-log-handler`` serializes rollover with a cross-process lock, but when another process holds that lock too long it raises this RuntimeError. Logging failures should not escape into Desktop chat output. r)rplatform isinstance RuntimeError_CONCURRENT_LOG_LOCK_TIMEOUTstr)r*s r'_is_windows_concurrent_log_lock_timeoutr2~s8  5 sL ) ) 5 (CHH 4r)openaizopenai._base_clienthttpxhttpcoreasynciohpackz hpack.hpackgrpcmodalurllib3zurllib3.connectionpool websocketscharset_normalizer markdown_it session_idc|t_dS)zSet the session ID for the current thread. All subsequent log records on this thread will include ``[session_id]`` in the formatted output. Call at the start of ``run_conversation()``. N_session_contextr>)r>s rset_session_contextrBs #-rcdt_dS)z,Clear the session ID for the current thread.Nr@rrrclear_session_contextrDs"&rctjtddrdSfd}d|_tj|dS)ubReplace the global LogRecord factory with one that adds ``session_tag``. Unlike a ``logging.Filter`` on a handler or logger, the record factory runs for EVERY record in the process — including records that propagate from child loggers and records handled by third-party handlers. This guarantees ``%(session_tag)s`` is always available in format strings, eliminating the KeyError that would occur if a handler used our format without having a ``_SessionFilter`` attached. Idempotent — checks for a marker attribute to avoid double-wrapping if the module is reloaded. _hermes_session_injectorFNcb|i|}ttdd}|rd|dnd|_|S)Nr>z []r)r rA session_tag)argskwargsrecordsidcurrent_factorys r_session_record_factoryz@_install_session_record_factory.._session_record_factorysG $1&11& d;;,/7[#[[[[R rT)logginggetLogRecordFactoryr rFsetLogRecordFactory)rOrNs @r_install_session_record_factoryrSsi133O :EBB 8<4  788888rcReZdZdZdeeddffd Zdejde fdZ xZ S)_ComponentFilterzOnly pass records whose logger name starts with one of *prefixes*. Used to route gateway-specific records to ``gateway.log`` while keeping ``agent.log`` as the catch-all. prefixesr+Ncptt||_dSr)super__init__tuple _prefixes)selfrV __class__s rrYz_ComponentFilter.__init__s* xrrLc@|j|jSr)name startswithr[r\rLs rfilterz_ComponentFilter.filters{%%dn555r) __name__ __module__ __qualname____doc__rr1rYrP LogRecordboolrb __classcell__r]s@rrUrUs~ )#)4))))))6W.6466666666rrU)gatewayhermes_pluginszplugins.platforms)agent run_agent model_tools batch_runner)tools) hermes_clicli)cron)zhermes_cli.web_serverzhermes_cli.pty_bridge tui_gatewayuvicorn)rkrmrqrsrtgui) hermes_home log_level max_size_mb backup_countmodeforcerxryrzr{r|r}c *|p t}|dz }|ddt\}} } |p|pd} t t | t j} |p| pddzdz} |p| pd}dd lm}t j }t||d z | | ||t t||d z t j d d|t |dkrIt||dz t jdd|tttd|dkrIt||dz t jdd|tttdtr|s|S|jt jks |j| kr|| t&D]3}t j |t j 4da|S)u@Configure the Hermes logging subsystem. Safe to call multiple times — the second call is a no-op unless *force* is ``True``. Parameters ---------- hermes_home Override for the Hermes home directory. Falls back to ``get_hermes_home()`` (profile-aware). log_level Minimum level for the ``agent.log`` file handler. Accepts any standard Python level name (``"DEBUG"``, ``"INFO"``, ``"WARNING"``). Defaults to ``"INFO"`` or the value from config.yaml ``logging.level``. max_size_mb Maximum size of each log file in megabytes before rotation. Defaults to 5 or the value from config.yaml ``logging.max_size_mb``. backup_count Number of rotated backup files to keep. Defaults to 3 or the value from config.yaml ``logging.backup_count``. mode Caller context: ``"cli"``, ``"gateway"``, ``"gui"``, ``"cron"``. When ``"gateway"``, an additional ``gateway.log`` file is created that receives only gateway-component records. When ``"gui"``, an additional ``gui.log`` file is created that receives dashboard and TUI-gateway component records. force Re-run setup even if it has already been called. Returns ------- Path The ``logs/`` directory where files are written. logsTparentsexist_okINFOirRedactingFormatterz agent.log)level max_bytesr{ formatterz errors.logi rkz gateway.logiP)rrr{r log_filterrwzgui.logi)r mkdir_read_logging_configupperr rPr agent.redactr getLogger_add_rotating_handler _LOG_FORMATWARNINGrUCOMPONENT_PREFIXES_logging_initializedrNOTSETsetLevel_NOISY_LOGGERS)rxryrzr{r|r}homelog_dir cfg_level cfg_max_size cfg_backup level_namerrbackupsrrootr_s r setup_loggingrsmX  +/++DVmG MM$M...+?*@*@'I|Z2y2F99;;J GZ 6 6E1 1T9D@I-j-AG0/////    D +$$[11  ,o!$$[11  y  m #,%((55'(:9(EFF     u}}  i ,&((55'(:5(ABB    E zW^##tzE'9'9 e::$((9999 Nrcddlm}tj}|jD]E}t |tjr)t |tst|ddrdSFtjt}| tj | |tdd|_|||jtj kr| tj t"D]3}tj| tj4tjd  tjdS) zEnable DEBUG-level console logging for ``--verbose`` / ``-v`` mode. Called by ``AIAgent.__init__()`` when ``verbose_logging=True``. rr_hermes_verboseFNz%H:%M:%S)datefmtTz rex-deploy)rrrPrhandlersr. StreamHandlerr r r)rDEBUG setFormatter_LOG_FORMAT_VERBOSEr addHandlerrrrr)rrhhandlerr_s rsetup_verbose_loggingr{sf 0/////    D] a. / /  1FY8Z8Z q+U33 #LNN33G W]### ++,?TTTUUU"GOOG zGM!! gm$$$::$((9999 l##,,W\:::::rceZdZdZfdZdZd dZd dZdej ddffd Z dej ddffd Z fd Z fd Z xZS)_ManagedRotatingFileHandleruRotatingFileHandler that ensures group-writable perms in managed mode AND survives external rotation. Two responsibilities: 1. In managed mode (NixOS), the stateDir uses setgid (2770) so new files inherit the hermes group. However, both ``_open()`` (initial creation) and ``doRollover()`` create files via ``open()``, which uses the process umask — typically 0022, producing 0644. This subclass applies ``chmod 0660`` after both operations so the gateway and interactive users can share log files. 2. ``RotatingFileHandler`` keeps an open file descriptor. If anything rotates the file *externally* (``logrotate``, manual ``mv``, another process rotating under us, a transient unlink), our fd keeps pointing at the renamed/unlinked inode and every subsequent write goes to ``gateway.log.1`` instead of ``gateway.log`` — silent log loss for the file every operator expects to read. Before each emit we ``stat`` ``baseFilename`` and compare it against the open stream's inode; on mismatch we reopen. This is the same pattern as stdlib ``WatchedFileHandler.reopenIfNeeded()``, adapted for rotating handlers. cddlm}||_tj|i|d|_d|_|dS)Nr) is_managed)hermes_cli.configr_managedrXrY _stat_dev _stat_ino_record_stream_stat)r\rJrKrr]s rrYz$_ManagedRotatingFileHandler.__init__sf000000"  $)&))))-(,   """""rcp|jr. tj|jddS#t$rYdSwxYwdS)Ni)roschmod baseFilenameOSError)r\s r_chmod_if_managedz-_ManagedRotatingFileHandler._chmod_if_managedsX =  *E22222      s % 33r+Nc tj|j}|j|jc|_|_dS#t$rd\|_|_YdSwxYw)zHSnapshot dev/ino of ``baseFilename`` so we can detect external rotation.)NNN)rstatrst_devst_inorrrr\sts rrz/_ManagedRotatingFileHandler._record_stream_stats` 8*++B-/Y *DNDNNN 8 8 8-7 *DNDNNNN 8s26AAc tj|j}n#t$r| |j|jn#t $rYnwxYwd|_ ||_|n#t $rYnwxYwYdSt$rYdSwxYw|j |j |j |j c|_ |_ dS|j |j f|j |j fkr |j|jn#t $rYnwxYwd|_ ||_|j |j c|_ |_ dS#t $rYdSwxYwdS)aaReopen the stream when ``baseFilename`` no longer matches our fd. Triggered when ``baseFilename`` was renamed (logrotate), unlinked, or replaced by a different inode. Silent + best-effort: any error falls back to the existing (possibly stale) stream so logging keeps working instead of dying on a stat failure. N)rrrFileNotFoundErrorr&r$r%_openrrrrrrrs r_reopen_if_externally_rotatedz9_ManagedRotatingFileHandler._reopen_if_externally_rotateds *++BB     ;*K%%'''    DK "jjll ((****     FF    FF  > !T^%;-/Y *DNDN F Iry !dndn%E E E ;*K%%'''    DK "jjll 13BI.     F Es B. AB. AB.A B. -B B. BB.BB.! B.-B.9 D D'&D'22E&& E43E4rLc|j$tj|jr|t |dSr)r&rpathexistsrrrXemit)r\rLr]s rrz _ManagedRotatingFileHandler.emitsQ ; "bgnnT5F&G&G "  . . 0 0 0  Vrctjd}t|rdSt|dS)uwSuppress the known Windows ``concurrent-log-handler`` lock timeout instead of printing a traceback. CLH's own ``emit()`` wraps its body in ``try/except Exception: self.handleError(record)``, so the ``"Cannot acquire lock after N attempts"`` RuntimeError raised in ``_do_lock()`` is caught inside CLH and routed here — it never propagates out of ``super().emit()``. This override is the single point where that timeout can be silenced before the stdlib handler prints it to stderr (which, under the Desktop slash-worker, is captured and surfaced into chat output).N)rexc_infor2rX handleError)r\rLr*r]s rrz'_ManagedRotatingFileHandler.handleError sHlnnQ 23 7 7  F F#####rcpt}||Sr)rXrr)r\r&r]s rrz!_ManagedRotatingFileHandler._opens+     rct||dSr)rX doRolloverrr)r\r]s rrz&_ManagedRotatingFileHandler.doRolloversE        """""rr+N)rcrdrerfrYrrrrPrgrrrrrirjs@rrrs 0#####8888////b7,$'"3$$$$$$$  #########rrzOptional[queue.SimpleQueue] _log_queue_queue_listener_queued_file_handlersc6eZdZdZdejdejfdZdS)_NonFormattingQueueHandleru``QueueHandler`` for an in-process queue. Stdlib ``prepare()`` formats the record and drops ``args``/``exc_info`` so it can be pickled to another process. Our queue is in-process, so we skip that and hand the target file handlers an unformatted record — they apply their own ``RedactingFormatter`` and component filters on the listener thread. We return a **shallow copy** rather than the original record: the same record is still owned by the emitting thread (and any synchronous handler on it, e.g. a ``StreamHandler``), which may format/mutate ``record.message`` while our listener thread reads it. Copying preserves ``msg``/``args``/ ``exc_info`` for the deferred format while removing the cross-thread mutation race on a shared object. rLr+c*tj|Sr)copyras rpreparez"_NonFormattingQueueHandler.prepareOsy   rN)rcrdrerfrPrgrrrrrr?sF  !g/!G4E!!!!!!rrcntdc}a|( |dS#t$rYdSwxYwdS)zAStop the listener assuming ``_queue_state_lock`` is already held.N)rstopr%listeners r_stop_queue_listener_lockedrSsV!0Ho  MMOOOOO    DD s $ 22cbt5tddddS#1swxYwYdS)zFlush and stop the background log listener (idempotent, thread-safe). This is the atexit hook, so it must acquire the state lock itself. N)_queue_state_lockrrrr_stop_queue_listenerr^sz &&#%%%&&&&&&&&&&&&&&&&&&s $((rct5tTtjat t}d|_t j|t |tt ttgtRddia ttst!jt$daddddS#1swxYwYdS)aRoute *handler* through the shared async queue instead of attaching it to *root* directly, so emitting threads never block on file I/O or the cross-process rotation lock. The ``QueueListener`` applies each handler's own level and filters on its worker thread.NTrespect_handler_level)rrqueue SimpleQueuer _hermes_queuerPrrrappendrrrstart_queue_atexit_registeredatexitregisterr)rqhs r_register_queued_handlerrgsE ,,  *,,J+J77B#B      * *2 . . .$$W---  &  " " "'  .   FJ   ' , O0 1 1 1'+ $/,,,,,,,,,,,,,,,,,,sC+DDDct5t}|(||ddddS#1swxYwYdS)uBlock until all queued records have been written, then resume. Draining is done by stopping the listener (which processes every pending record before joining) and restarting it. Used by tests that read a log file right after emitting to it. NOTE: ``stop()`` joins the worker thread, so this blocks until the queue is empty. Do NOT call this on a hard-exit path where the listener may be wedged on the rotation lock — use ``drain_log_queue()`` there instead, which bounds the wait. N)rrrrrs rflush_log_queuers "   MMOOO NN    s2AA A ?timeoutctdSdfd }tj|dd}|||dS)uBest-effort, time-bounded drain for hard-exit paths (no restart). Unlike ``flush_log_queue()``, this stops the listener WITHOUT restarting it (the process is about to exit) and bounds the drain: if the listener's worker thread is wedged on the cross-process rotation lock — the very failure this async-logging change exists to survive — an unbounded ``stop()``/join would re-freeze the shutdown path. We run ``stop()`` on a throwaway thread and only wait ``timeout`` seconds for it; if it hasn't drained by then we abandon the last few records and let ``os._exit`` proceed. Availability beats the last log line when the disk is already wedged. Nr+cT dS#t$rYdSwxYwr)rr%rsr_drainzdrain_log_queue.._drains;  MMOOOOO    DD s  ''zhermes-log-drainT)targetr_daemonr)r threadingThreadrjoin)rrtrs @rdrain_log_queuersmH -?MMMAGGIIIFF7OOOOOrc*ttS)zReturn the live rotating file handlers. They are attached to the async ``QueueListener`` rather than the root logger, so callers/tests must use this instead of scanning ``logging.getLogger().handlers``.)listrrrrrotating_file_handlersrs % & &&rct5ttj}t |jD](}t |ddr||)t tD]'} | #t$rY$wxYwt da ddddS#1swxYwYdS)zETear down the async logging queue + listener (test-isolation helper).rFN) rrrPrrrr removeHandlerrr$r%clearr)rrs r_reset_queued_handlersrs-   #%%% ""dm$$ & &Aq/511 &""1%%%+,,  A      ##%%%                   s6A5C >BC  B C B  C  CC)rloggerrrrrrc|}tD]N}t|tr7t t |dd|krdSO|jddtt|||d} | || ||| |t| dS)aAdd a ``RotatingFileHandler`` to *logger*, skipping if one already exists for the same resolved file path (idempotent). Parameters ---------- log_filter Optional filter to attach to the handler (e.g. ``_ComponentFilter`` for gateway.log). rrNTrr)maxBytes backupCountr)resolverr.r rr parentrrr1rr addFilterr) rrrrr{rrresolvedexistingrs rrrs&||~~H) x!4 5 5 WX~r::;;CCEEQQ FFKdT222) D I<G U ###*%%%W%%%%%rc ddlm}t}|rt |dd5}||pi}dddn #1swxYwY ddlm}||}n#t$rYnwxYw| di}t|tr>| d | d | d fSn#t$rYnwxYwd S) uBest-effort read of ``logging.*`` from config.yaml. Returns ``(level, max_size_mb, backup_count)`` — any may be ``None``. r)fast_safe_loadrr)rN) managed_scoperPrrzr{)NNN) utilsrr ropenrrrapply_managed_overlayr%getr.dict)r config_pathfcfgrlog_cfgs rrrs  ((((((%''      k3999 .Q$nQ''-2 . . . . . . . . . . . . . . . 444444#99#>>    ggi,,G'4(( KK((KK ..KK//       sY:C8A C8AC8AC8"A>=C8> B C8 B  A+C88 DDr)r)Drfrrr"rPrrrrlogging.handlersrrpathlibrtypingrrr-concurrent_log_handlerr r hermes_constantsr r rlocalrArrr)r0 BaseExceptionrhr2rr1rBrDrSFilterrUrintrrrr__annotations__rrrrLockrrrrHandlerrrfloatrrrLogger Formatterrrrrrr*s:  88888888%%%%%%%%0<7544444>======= #9?$$ O [D G 1E $    ",-C-D----''''9999: !!! 6 6 6 6 6w~ 6 6 6,B B   2#'#!%"&uuu$u}u# u 3- u 3- u u uuuup;;;;HD#D#D#D#D#"5D#D#D#f-1 )000+/-(/// t  #IN$$!!!!!!!!(&&&&,go,$,,,,@&UT8'''''4,0&&&&&& N&& &&  &&  &&  && &&(&& &&&&&&&&Rr