#jͲ TdZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZmZmZdd lmZmZmZmZmZmZmZdd lmZmZdd lmZeje Z!d Z"ej#Z$d%de%de%fdZ&dede%ddfdZ'de%de(fdZ)d&de%de%de%dzddfdZ* d'de%de%dzde%dzde%dzddf dZ+ d(de%dzde%dzde,e%dze%dzffdZ-dede%de%de(de%ddf dZ. d)de%de(de%dzde%ddf d Z/ d*de%de%de%dzddfd!Z0 d+de%d#e%de%dzddfd$Z1dS),a5 Database migration management using Alembic. This module provides programmatic access to run database migrations on application startup. It is designed to be safe for concurrent execution using PostgreSQL advisory locks to coordinate between distributed workers. Supports multi-tenant schema isolation: migrations can target a specific PostgreSQL schema, allowing each tenant to have isolated tables. Important: All migrations must be backward-compatible to allow safe rolling deployments. No alembic.ini required - all configuration is done programmatically. N)Path)command)Config)ResolutionError) Connection create_enginetext)bootstrap_extensiondetect_vector_extensionindex_type_keywordindex_using_clauseminimum_rows_for_indexshould_defer_index_creationuses_per_bank_vector_indexes) is_oracle_url to_libpq_url)mask_network_locationi[pgvectorvector_extensionreturnc"t||S)zJValidate configured vector extension and preserve Azure DiskANN detection.)r )connrs `/home/rurouni/.hermes/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/migrations.py_detect_vector_extensionr7s "4)9 : ::r schema_namec 4|tdd|i}|dd}|D]G}|ddd}|td|d|dHdS) zNDrop per-bank partial memory_units vector indexes after global ScaNN is ready.z SELECT indexname FROM pg_indexes WHERE schemaname = :schema_name AND tablename = 'memory_units' AND indexname LIKE 'idx_mu_emb_%' AND indexdef LIKE '%embedding%' r"z""rzDROP INDEX IF EXISTS "z"."N)executer fetchallreplace)rrrows safe_schemarow safe_indexs r_drop_per_bank_vector_indexesr'<s <<      $  hjj %%c400KSSV^^C..  TP;PP:PPPQQRRRRSSrschemactj|dd}t|ddzS)z{ Generate a unique advisory lock ID for a schema. Uses hash of schema name to create a deterministic lock ID. Nbig) byteorderl)hashlibsha256encodedigestint from_bytes)r( hash_bytess r_get_schema_lock_idr4PsK 007799"1"=J >>*> 6 6% @@r database_urlscript_locationc|pd}td|dtdt|td|t}|d||d||dd |d d |r|d | t 5t j|d dddn #1swxYwYn5#t$r(}t d|Yd}~dSd}~wwxYwtd|ddS)z Internal function to run migrations without locking. Args: database_url: SQLAlchemy database URL script_location: Path to alembic scripts schema: Target schema (None for default/public) publicz0Running database migrations to head for schema 'z'...zDatabase URL: zScript location: r6zsqlalchemy.urlprepend_sys_path.path_separatoros target_schemaheadsNzDatabase is at a newer migration revision than this code version knows about. This is expected during rolling deployments. Skipping migrations. Error: z7Database migrations completed successfully for schema '') loggerinforrset_main_option _alembic_lockrupgraderwarning)r5r6r(r alembic_cfges r_run_migrations_internalrH\s$HK KKT;TTTUUU KKF!6|!D!DFFGGG KK5O55666((K 1?CCC 0,??? 2C888 0$777=##OV<<<   2 2 OK 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2      \XY \ \      KKX+XXXYYYYYs<)D0D DDDDD E(E  Emigration_database_urlc  |p|}t|r|nt|} |+ttj}t |dz }t|}|std|dt|rt|||dS|rt|nt}|pd} t|} | 5} td| d|d  | t!d |d } | rn)| t'jd d| td td| t!d} | r,| d}|dkrtdnDtd|d | t!d| t!d| t!d| tdn#t0$r@}td|d|d| Yd}~nJd}~wwxYwtd | t!d| t!d| tdn#t0$r}| | t!d}|r%td|d d!n.td"|d#t7d$|Yd}~nd}~wwxYwt9jd%d&}t?| || t|||| t!d'|d td(nE#| t!d'|d td(wxYw ddddS#1swxYwYdS#t$rtd|t@$rA}td)|j!d *t7d+|j!|d}~wt0$r4}td,|d *t7d-|d}~wwxYw).ar Run database migrations to the latest version using programmatic Alembic configuration. This function is safe to call from multiple distributed workers simultaneously: - Uses PostgreSQL advisory lock to ensure only one worker runs migrations at a time - Other workers wait for the lock, then verify migrations are complete - If schema is already up-to-date, this is a fast no-op Supports multi-tenant schema isolation: when a schema is specified, migrations run in that schema instead of public. This allows tenant extensions to provision new tenant schemas with their own isolated tables. Args: database_url: SQLAlchemy database URL (e.g., "postgresql://user:pass@host/db") script_location: Path to alembic migrations directory (e.g., "/path/to/alembic"). If None, defaults to hindsight-api/alembic directory. schema: Target PostgreSQL schema name. If None, uses default (public). When specified, creates the schema if needed and runs migrations there. Raises: RuntimeError: If migrations fail to complete FileNotFoundError: If script_location doesn't exist Example: # Using default location and public schema run_migrations("postgresql://user:pass@host/db") # Run migrations for a specific tenant schema run_migrations("postgresql://user:pass@host/db", schema="tenant_acme") # Using custom location (when importing from another project) run_migrations( "postgresql://user:pass@host/db", script_location="/path/to/copied/_alembic" ) Nalembicz%Alembic script location not found at z$. Database migrations cannot be run.)r(r8z.Acquiring migration advisory lock for schema 'z' (id=z)...TzSELECT pg_try_advisory_lock()g?z Migration advisory lock acquiredz+Checking pgvector extension availability...zrSELECT extname, nspname FROM pg_extension e JOIN pg_namespace n ON e.extnamespace = n.oid WHERE extname = 'vector'r z8pgvector extension found in public schema - ready to usez$pgvector extension found in schema 'z0' instead of 'public'. Attempting to relocate...zDROP EXTENSION vector CASCADEzSET search_path TO publiczCREATE EXTENSION vectorz-pgvector extension relocated to public schemaz8Could not relocate pgvector extension to public schema: z . Continuing with extension in 'z ' schema.z6pgvector extension not found, attempting to install...z-pgvector extension installed in public schemaziSELECT nspname FROM pg_extension e JOIN pg_namespace n ON e.extnamespace = n.oid WHERE extname = 'vector'zTCould not install pgvector extension (permission denied?), but extension exists in 'rz' schema. Continuing...z=pgvector extension is not installed and cannot be installed: zy. Please ensure pgvector is installed by a database administrator. See: https://github.com/pgvector/pgvector#installationzbpgvector extension is required but not installed. Please install it with: CREATE EXTENSION vector;HINDSIGHT_API_VECTOR_EXTENSIONrzSELECT pg_advisory_unlock(z Migration advisory lock releasedz%Alembic called sys.exit() with code: )exc_infoz)Database migration failed with exit code z#Failed to run database migrations: zDatabase migration failed)"rrr__file__parentstrexistsFileNotFoundErrorrHr4MIGRATION_LOCK_IDrconnectr@debugr r scalarcommittimesleepfetchonerArE Exceptionrollbackerror RuntimeErrorr<getenvlowerr SystemExitcode)r5r6r(rIraw_url migration_url package_dir script_pathlock_idrengineracquired ext_check ext_schemarG ext_recheckrs rrun_migrationsrns`%4 G -W55PGG<;P;PMd?  "x../K!+ "9::O?++ !!## #mmmm   ' '  $]OF S S S S F28N%f---=N( &}-- ^^  m A LLj+jj]djjj k k k <<-VG-V-V-V(W(WXX__aa  3  KKMMM LL; < < <[ A JKKK!LL3 (** ;%!*1J!X-- $^____9:999 , LL.M)N)NOOO LL.I)J)JKKK LL.G)H)HIII KKMMM"KK(WXXXX(,,,"NN!W[\!W!WAK!W!W!W!MMOOOOOOOO ,KK XYYY % T*E%F%FGGG T*C%D%DEEE  $STTTT$%%% &*ll !;'' #(** $'%"NN!d1>> NNN677Q>?sBU,?U+B5U!BS>;BKS> L5LS>LS>9A2N,+S>, Q6BQS>QA S>=AU>AUU UUUUU1X >>>>>222222,,,,,,  9%ABBL  NNq   :|L99:: ^^   9&00<|d | d?t/|d@ |dkr| dkrt!||f|tdA|ddddS#1swxYwYdS)Ba Ensure the vector indexes match the configured vector extension. This function checks the current vector index type in the database and adjusts it if necessary: - If index type matches configured extension: no action needed - If they differ and tables are empty: drop old indexes, recreate with new type - If they differ and tables have data: raise error with migration guidance Args: database_url: SQLAlchemy database URL vector_extension: Configured vector extension ("pgvector", "vchord", "pgvectorscale", or "scann") schema: Target PostgreSQL schema name (None for public) Raises: RuntimeError: If extension mismatch with existing data r8zTarget vector extension: ))ridx_memory_units_embedding) learningsidx_learnings_embedding)pinned_reflections idx_pinned_reflections_embedding SELECT EXISTS ( SELECT 1 FROM information_schema.tables WHERE table_schema = :schema AND table_name = :table_name ) r(rTable  does not exist in schema ' ', skippingrr:rz SELECT indexdef FROM pg_indexes WHERE schemaname = :schema AND tablename = :table_name AND indexname LIKE :index_pattern z %embedding%)r(r index_patternra" SELECT COUNT(*) FROM pg_indexes WHERE schemaname = :schema AND tablename = :table_name AND indexname LIKE 'idx_mu_emb_%' rzNo global embedding index on z, but uI per-bank partial vector indexes exist — skipping global index creationzNo embedding index found for z, will create it if safeNscanndiskannvchordrqhnswzUnknown index type for rzIndex type mismatch on z : current=z , target=zIndex type OK for z/All vector indexes match configured extension: , c&g|]\}}}|d|dS(z rows)).0rcount_s r z+ensure_vector_extension..s/#e#e#equ$<$-I-I1$ DDDKKMMM!  LLW:WW X X X QtPtPtPtPtPtPtPtPV  #e#eTd#e#e#effJ!1!!4Q!7 +$"    c$&899 s7IssO`ss6@ss8Css + ssEP ss _o sss   BjBBCCC?P0 A0 A ;J L)*:yAA 5jAA =   W L LL LLMMM T"T+"T"T "T"TUUVVVZ'' LL +*EE  &((T!1!1&kykkjkkk KKM$5MMMM N N N LL0:#&0( 33   W$$~)E)E-dK@@@  N*NNOOOitPtPtPtPtPtPtPtPtPtPtPtPtPtPtPtPtPtPsLU6H%U66U:=U:nativetext_search_extensionc. |pd}tt|}|5}ddg}|dkrd}d}n|dkrd}d}nd }d }g} g} |D]} |t d || d } | s"t d | d|d_|t d|| d } | s7t d| d| | ddf| d}|t d|| d }|r|dnd}||k}|r||knd}|r|st d| d|d|d|d|d | | ||f|t d|d| }|dkr| | |ft d| d |d!|| s+t d"| ddddS| rld# d$| D}| dd}|d krd%}n|dkrd}n |dkrd}nd&}td'|d(|d)|d*|d+|d,|d- t d.|| D]\} }}|r]t d/|d0| |t d1|d2| dd3d4|rIt d/|d5| |t d6|d| d7|dkrt d8| |t d6|d| d9t d:| |t d;| dd3d<|d| d=\|dkrt d>| |t d6|d| d?t d:| | dkrd@}ndA}|t d;| dd3d<|d| dB|dC t dD| | dkrdE}ndF}|t dG|d| dH|dIt dJ| |t d;| dd3d<|d| dK|t dL|ddddS#1swxYwYdS)Ma Ensure the text search columns and indexes match the configured extension. This function checks the current search_vector column type and index type in the database and adjusts them if necessary: - If they match configured extension: no action needed - If they differ and tables are empty: drop old column/index, recreate with new type - If they differ and tables have data: raise error with migration guidance Args: database_url: SQLAlchemy database URL text_search_extension: Configured text search extension ("native" or "vchord") schema: Target PostgreSQL schema name (None for public) Raises: RuntimeError: If extension mismatch with existing data r8r reflectionsr bm25vectorbm25 pg_textsearchr tsvectorginrrrrra SELECT data_type, udt_name FROM information_schema.columns WHERE table_schema = :schema AND table_name = :table_name AND column_name = 'search_vector' z"No search_vector column found for z, will create itNr av SELECT am.amname FROM pg_indexes pi JOIN pg_class c ON c.relname = pi.indexname JOIN pg_am am ON am.oid = c.relam WHERE pi.schemaname = :schema AND pi.tablename = :table_name AND pi.indexname LIKE '%text_search%' rFzText search mismatch on z : column=z (want z ), index=rLrr:zText search OK for r/z.s-#b#b#bu$<$OM M  :J(*:  P(8PPJPPQQQ .9@J@R@RSVX[@\@\  r Q(8QQZQQRRR T"o"o"oz"o"o"oppqqq%00 IZIIJJJ s ssjssstt  BjBBCCC *4*<*rs\"  !!!!!!33333366666666660///////((((((  8 $ $   ;;S;#;;;; S SSSSSS( A A A A A A4Z4Z34Z4ZcTXj4Zdh4Z4Z4Z4Zr#')- Y?Y?Y?4ZY? $JY? $J Y?  Y?Y?Y?Y?zDH88*869Dj8 3:sTz !"8888vmb mbmbmb mb  mb  mbmbmbmbf& /o/o/o/o $J/o /o  /o/o/o/oh'MPMPMPMP $JMP MPMPMPMPd"*dUdUdUdU $JdU dUdUdUdUdUdUr