IjxH UdZddlZddlZddlZddlZddlZddlZddlZddlZ ddl Z ddl Z ddl m Z ej ej eZeej vrej deddlmZej dZej dZej dZej dZej d Zej ed Zejed ejeejd dej dZ!dZ"da#e$dze%d<de&fdZ'dZ(ej dZ)de*de+fdZ,d8de*de*de*de*fdZ-de+fdZ.de/fdZ0d e*de*fd!Z1d"e*de&fd#Z2de+dzfd$Z3d e*de&fd%Z4de+de&fd&Z5de+de&fd'Z6d9d)e+d*e*fd+Z7d:d-e$d.e$de/e&e8ffd/Z9d)e+de&fd0Z:d1Z;d2Z$r@Z?e!@d4d 5e"d6e?ejAd7YdZ?[?dSdZ?[?wwxYwdS);a Auto-switch llama-server service when Hermes config model changes. Triggered by systemd path unit (hermes-config.path) on config.yaml changes. Uses flock-based lock file to prevent concurrent invocations from racing. Uses state file guard to prevent re-trigger loops. Performs rollback if the target service fails to start. Logs to ~/.hermes/logs/model-switch.log for troubleshooting. Single source of truth for model definitions: models.py N)datetime)MODELSz~/.hermes/config.yamlz~/.hermes/.last-model-statez~/.hermes/.model-switch-notifyz~/.hermes/.model-switch.lockz~/.hermes/logszmodel-switch.logTexist_okz'%(asctime)s [%(levelname)s] %(message)sz%Y-%m-%d %H:%M:%S)filenamelevelformatdatefmtz model-switchc$t|dS)NT)flush)print)msgs 2/home/rurouni/.hermes/scripts/auto-model-switch.pyr2sE#T***_LOCK_FDreturnc tjttjtjzd}t j|t jt jz|a tj |dtj |tj d dS#ttf$r/t$dt!vrtj|YdSwxYw)zAAcquire an exclusive lock on LOCK_FILE. Returns True if acquired.ir TNfdF)osopen LOCK_FILEO_CREATO_RDWRfcntlflockLOCK_EXLOCK_NBr ftruncatewritegetpidencodeIOErrorOSErrorlocalsclose)rs r acquire_lockr(9s  WY RY 6 > > B  5666 R  '''..00111t W    0 0 HRLLLuusB0B44>( # # Ih       27>>( # # Ih     s=599F%.AE F%EF%EF%%6Gctjts%tdtdddSt t}|dd|dddS)z7Read model.provider and model.default from config.yaml.zConfig file not found: %sN)rGrHzmodel.providerr2z model.default)rrZrcCONFIGlogwarningrFget)parseds rget_config_settingsrosz 7>>& ! !1 /888 4000 v & &FJJ/44OR00  rcrtjtsdSt t5}|d}dddn #1swxYwY|r|dnd}t|dkr|dnd}||fS)z+Read last-known-GOOD state from state file.)NNrNr) rrZrc STATE_FILErreadr9splitr6)r=linesrHrGs rget_last_good_statervs 7>>* % %| j  -Q  &&t,,--------------- 'E!HH4Eu::>>uQxxtH 8 s:BBB model_namec|d|d}ttd5}||dddn #1swxYwYtd||dS)z%Persist the last good model+provider.rrQNz&Good state saved: provider=%s model=%s)rrrr!rkdebug)rwrGcontentr=s rsave_good_stater{s++x+++G j#  ! II6*MMMMMsAAAmodel_idctjtstddS t t5}tj|}dddn #1swxYwYd|dddD}||vS#tj tf$r&}t d|Yd}~dSd}~wwxYw) z=Check if an OpenRouter model is in the curated override list.u6OpenRouter override file not found — cannot validateTNcg|] }|d S)idr,).0ms r z/is_allowed_openrouter_model..sNNNq1T7NNNr providers openroutermodelsz'Failed to parse OpenRouter override: %s) rrZrcOPENROUTER_OVERRIDErkrlrjsonloadJSONDecodeErrorKeyErrorerror)r|r=dataallowedes ris_allowed_openrouter_modelrs 7>>- . . LMMMt % & & !9Q<A;?$B$$C :CC ctD]Z}|drtjdd|dgddd}|jdkr|cS[d S) zPFind which model service is currently active (systemd), skipping preserved ones.preserve systemctlz is-activeserviceTcapture_outputrPtimeoutactiveN)rrm subprocessrunstdoutr9)rrs rget_active_servicers  55     N +q| 4dA    8>>  x ' 'HHH ( 4rcFtfdtD S)zCCheck if a model_name refers to a cloud model (not in MODELS list).c30K|]}|dkVdSrwNr,)rrrws r z!is_cloud_model..s,AAQ1\?j0AAAAAAr)anyr)rws`ris_cloud_modelrs)AAAA&AAAAA AArctd|d|dtjddd|dgddd }|jd kr"td |dnGtd |d|j|jd d|jd kS)z?Stop a single model's systemd service. Returns True on success.zStopping %s (%s)...namersudorstopTrru ✅ Stopped %su! ⚠️ Stop %s returned %d: %sN)rkinforr returncoderlstderrr9rHrs r stop_servicersHH "E&M53CDDD feI&67$   A |q #U6]3333 76]AL!(..2B2B4C42H J J J <1 rctd|d|dtjddd|dgddd }|jd kr"td |dnGtd |d|j|jd d|jd kS)z9Start a model's systemd service. Returns True on success.zStarting %s (%s)...rrrrstartTi,rru ✅ Started %su ❌ Start %s failed (%d): %sN)rkrrrrrrr9rs r start_servicersHH "E&M53CDDD guY'78$   A |q #U6]3333 2=!,0@0@#0F H H H <1 rswitchedtargetactionc d} tjgdddd}|jdkrS|jd}t |dkr|dd |d d }n#t$rYnwxYwtj d }d |d|dd|dd|d| ddd|d }tj tjt dt#t d5}||dzddddS#1swxYwYdS)z?Write a one-shot notification file for the watchdog to deliver.?)z nvidia-smiz$--query-gpu=memory.used,memory.totalz--format=csv,noheader,nounitsTrrrz, z MB / rqz MBz %I:%M:%S %puO🔀 **Model Switch** ━━━━━━━━━━━━━━━ **Event:** z **Model:** rz **Port:** portz **VRAM:** z **Context:** contextz K **Time:** z PDTrrQrN)rrrrr9rtr6 ExceptionrnowstrftimermrmakedirsrZr[ NOTIFY_FILErr!)rrvramrpartstsrr=s rwrite_notificationrs D  N . . .dA    <1  HNN$$**400E5zzQ(77%(777        / /B   Vn  F^         9c22      K ,,t<<<< k3  1 d s$A8A== B  B 5EE"E<r timeout_seccd|d}tj}t|D]} tjtj|d}|jdkrdtj|z fcSn#t$rYnwxYwtjddtj|z fS) zEWait for the model API to respond. Returns (success, seconds_waited).http://127.0.0.1:z /v1/modelsr)rrTrqF) timerangeurllibrequesturlopenRequeststatusrsleep)rrurlrrBresps rwait_for_modelrs .d . . .C IKKE ;   >))&.*@*@*E*Eq)QQD{c!!TY[[500000"    D  1 $)++% %%sA B BBc t}|r5|d|dkr#td|ddS|}|r|ddkr|ddkr{|drt d|ddnht d |dd t |t d t jd n|std |ddkret d|dd t|}|rt d|ddt|dd\}}|rt d|dd|ddnt d|dd|ddtd|r|dnd|d|rdnd|d |dd!}tt|dd"|d#|t|d$dSt d%|dt d&|rt d'|dd t|}|rtd(|dtt|dd"|d#d |dd!t|d)t d*|dn:td+|dt d,|dd-d.St d/t d0d.St d1|d#d2dS)3zStop the currently active service, start target. Does NOT write config.yaml (caller already did). Only rollback writes config. ru!Already on %s — nothing to swapTrGlocalru 🛡 Preserving z (background service)u ⏹ Stopping ...u# 💤 Waiting 3s for VRAM drain...zNo active service to stopu ▶️ Starting u ⏳ Waiting for z to respond...rr)ru ✅ z responded in z.1fsu ⚠️ z API not responding after z.0fus — continuingu3Switch complete: %s → %s (API ready: %s in %.1fs)z(none)yesnor/v1 custom:localrwru ❌ Failed to start u%Switch FAILED — attempting rollbacku ↩️ Rolling back to z!Rollback successful: restarted %srollbacku ✅ Rolled back to u.Rollback FAILED — %s could not be restarted!u ❌ Rollback failed! z not running!Fz#No previous service to roll back tou- ⚠️ No previous service to roll back tou ☁️ Cloud model u — no service swap needed)rrkrrm_echorrrrrrhrjrrcritical)rrpreviousokloadedelapsedcorrect_base_url rollback_oks rswap_tor(s ! !F&.F6N22 4fVnEEEtH .&$//F:4F'4Q4Q ::j ! !  MvMMM N N N N 8VF^888 9 9 9  7 8 8 8 JqMMMM . ,---jW$$ 7F6N777888 6 " "   Evf~EEE F F F,VF^LLLOFG lKvf~KKWKKKKLLLLj6&>jjWjjjjkkk HHJ)1?Xf%%x$.UU$ 9 9 9 G6&>FFF  VVZZ N%K%K#L13C E E E vz 2 2 24 7vf~77888 9:::   Dx/?DDD E E E'11K QNOOO j.)Q)Q!),!7!JXf5E!J!J!JLLL#8Z888@hv.>@@AAAA MxX^O_```O0@OOOPPP5 II; < < < A B B B5 T6,#7 T T TUUU 4rcts.tdtjd t t dS#t wxYw)Nu(Lock held by another process — exitingr)r(rkrysysexit _main_implr-r,rrmainrqsW >> <===    s AA,c  t}|d |d} stddS|dkr:t\}}t st d dtd |rL| krFt d|td |tt|pd|nEd }t d |td |ttd |t}t|d|ddSt |td dSt r-td t |dSt fdtDd}|std dSt\}}| krt fdtDd}|rEt}|r/|d|dkrtd dS|r)td|d|dntd|dt d|ddt|}|rvd|dd} tt|dd |d| t#|dt |t d|dnt d|ddStd dStd!|dt d"|dd#t%|} | r*t |t d$|ddSdS)%NrHrGuNo model in config — exitingru ❌ 'z/' is NOT in your curated OpenRouter model list!z2BLOCKED: OpenRouter model '%s' not in allowed listu ↩️ Reverting to z'Reverting config to last good model: %szQwen3.6-35B-A3B-UD-Q4_K_S.ggufu5 ↩️ No last good state — reverting to local: z&Reverting config to local fallback: %srzOpenRouter model validated: %su4Config set to cloud model: %s — no service to swapc34K|]}|dk|VdSrr,rr config_models rrz_main_impl..s1HH,<(G(G1(G(G(G(GHHru(Unknown model in config: %s — ignoringc34K|]}|dk|VdSrr,rs rrz_main_impl..s1LLQAlO|,K,Kq,K,K,K,KLLrru2State unchanged (%s) — service healthy, skippingu2State says %s but %s is active — fixing mismatchu0State says %s but service is dead — restartingu 🔄 u is dead — restarting...rrrrw restartedu ✅ Restarted u ❌ Failed to restart u!State unchanged (%s) — skippingz!Config changed to local model: %su 🔄 Auto-switching to ru ✅ Switched to )rorkrrvrrrlrhrjr{rnextrrryrrmrr) settingsconfig_providerlast_good_modellast_good_providerfallbackrrBrr correct_urlr:rs @rrr{sf"$$HG$Lz*O  1222,&&.A.C.C++*<88  YLYYY Z Z Z KKLl [ [ [ C?l#B#BAAABBBBOTTT );)K|_]]]];XhXXYYYA8LLL BBB*,,H HW-x /C D D D F o666 1<@@@l## GVVV o666HHHHfHHH$ O OF  > MMM-..OQ,&&LLLL&LLLdSS  '))F &.F6N:: NP\]]]  P"6NF6N<<<< NPVW]P^___Ivf~IIIJJJ"6**G"IfVn"I"I"IK$VVZZ N-S-S%+L%9;HHH&v{;;;#L/BBB334444455r__main__z$Unhandled error in auto-model-switch)exc_infou ❌ Auto-switch error: rq)r2)r)r)B__doc__rrrrloggingrr\urllib.requestrrrUrrZr[abspath__file__ _script_dirinsertrr expanduserrjrrrrLOG_DIRjoinLOG_FILEr basicConfigINFO getLoggerrkrrint__annotations__boolr(r-rstrrXrFrhrotuplervr{rrrrrrfloatrrrr__name__rrrrr,rrrs     goobgooh7788 chHOOA{###   3 4 4 W   = > > g  !ABB G  = > > '  - . . 7<<!3 4 4  Gd####  , 4   g'' +*#*d"   "g(()STTB  s c #     6 T    UNNsNNNN # $     D4K    BsBtBBBB        $    tS> & & &3 &dEk8J & & & & EDETEEEERP5P5P5f z   84 HHH -!--... s H%%I**5I%%I*