#j>RUdZddlZddlZddlZejdrejdndZddlZddl Z ddl m Z ddl m Z mZddlmZddlmZddlmZdd lmZmZdd lmZe rddlZd efd Zd ZdZdZde d efdZ!ej"e#Z$da%d!dedefdZ&dZ'GddZ(Gdde(Z)Gdde(Z*e)a+e(e,d<d e(fdZ-d e*fd Z.dS)"a OpenTelemetry metrics instrumentation for Hindsight API. This module provides metrics for: - Operation latency (retain, recall, reflect) with percentiles - Token usage (input/output) per operation - Per-bank granularity via labels - LLM call latency and token usage with scope dimension - HTTP request metrics (latency, count by endpoint/method/status) - Process metrics (CPU, memory, file descriptors, threads) - Database connection pool metrics Nresource)contextmanager) TYPE_CHECKINGCallable)metrics)PrometheusMetricReader) MeterProvider)"ExplicitBucketHistogramAggregationView)Resourcereturnc"ddlm}|S)z>Get current tenant (schema) from context for metrics labeling.rget_current_schema)"hindsight_api.engine.memory_enginerrs ]/home/rurouni/.hermes/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/metrics.py _get_tenantr"s&FEEEEE    )皙???g??@@@g@$@.@g4@>@N@^@) rrrrrrrrrrrr ) g{Gzt?g{Gz?g?g?rrrrg@rrr token_countcf|dkrdS|dkrdS|dkrdS|dkrdS|d krd S|d krd Sd S)a4 Convert a token count to a bucket label for use as a dimension. This allows analyzing token usage patterns without high-cardinality issues. Buckets: - "0-100": Very small requests/responses - "100-500": Small requests/responses - "500-1k": Medium requests/responses - "1k-5k": Large requests/responses - "5k-10k": Very large requests/responses - "10k-50k": Huge requests/responses - "50k+": Extremely large requests/responses Args: token_count: Number of tokens Returns: Bucket label string dz0-100iz100-500iz500-1kiz1k-5ki'z5k-10kiPz10k-50kz50k+)r!s rget_token_bucketr%5si*Sw s  y t  x t  w u  x u  yvr hindsight-api1.0.0 service_nameservice_versionctj||d}t}tdt t }tdt t }tdt t}t||g|||g}tj |tj ta |S)aT Initialize OpenTelemetry metrics with Prometheus exporter. This should be called once during application startup. Args: service_name: Name of the service for resource attributes service_version: Version of the service Returns: PrometheusMetricReader instance (for accessing metrics endpoint) )z service.namezservice.versionhindsight.operation.duration) boundaries)instrument_name aggregationhindsight.llm.durationhindsight.http.duration)rmetric_readersviews)r createrr r DURATION_BUCKETSLLM_DURATION_BUCKETSHTTP_DURATION_BUCKETSr rset_meter_provider get_meter__name___meter)r(r)rprometheus_reader duration_viewllm_duration_viewhttp_duration_viewproviders rinitialize_metricsr@`s (.  H/0066BRSSSM 06BVWWW 16BWXXX )*/1CDH x((( x ( (F rc<ttdtS)zGet the global meter instance.Nz9Metrics not initialized. Call initialize_metrics() first.)r: RuntimeErrorr$rrr8r8s ~VWWW MrceZdZdZe ddededededzdedzf d Z dd ed ededededede fdZ ededede geffdZ ddZ dS)MetricsCollectorBasez"Base class for metrics collectors.apiN operationbank_idsourcebudget max_tokensct)z8Context manager to record operation duration and status.NotImplementedErrorselfrFrGrHrIrJs rrecord_operationz%MetricsCollectorBase.record_operations "!rrTr?modelscopeduration input_tokens output_tokenssuccessct) Record metrics for an LLM call. Args: provider: LLM provider name (openai, anthropic, gemini, groq, ollama, lmstudio) model: Model name scope: Scope identifier (e.g., "memory", "reflect", "consolidation") duration: Call duration in seconds input_tokens: Number of input/prompt tokens output_tokens: Number of output/completion tokens success: Whether the call was successful rLrOr?rQrRrSrTrUrVs rrecord_llm_callz$MetricsCollectorBase.record_llm_calls ,"!rmethodendpointstatus_code_getterct)z/Context manager to record HTTP request metrics.rLrOr[r\r]s rrecord_http_requestz(MetricsCollectorBase.record_http_requests "!rpool asyncpg.PoolcdS)z-Set the database pool for metrics collection.Nr$rOras r set_db_poolz MetricsCollectorBase.set_db_pools rrENNrrTrarb)r9 __module__ __qualname____doc__rstrintrPfloatboolrZrr`rer$rrrDrDs>,, !!% " " " " " d " $J " " "^ """""" "  "  """"""0"#""RZ[]_b[bRc"""^"      rrDceZdZdZe ddededededzdedzf d Z dd ed ededededede fdZ ededede geffdZ dS)NoOpMetricsCollectorzJNo-op metrics collector that does nothing. Used when metrics are disabled.rENrFrGrHrIrJc#KdVdS)zNo-op context manager.Nr$rNs rrPz%NoOpMetricsCollector.record_operations rrTr?rQrRrSrTrUrVcdS)zNo-op LLM call recording.Nr$rYs rrZz$NoOpMetricsCollector.record_llm_calls  rr[r\r]c#KdVdS)zNo-op HTTP request recording.Nr$r_s rr`z(NoOpMetricsCollector.record_http_requests rrfrg) r9rirjrkrrlrmrPrnrorZrr`r$rrrqrqs*TT !!%      d  $J   ^ "                    #RZ[]_b[bRc^rrqceZdZdZdZe ddededededzd edzf d Z dd ededede dedede fdZ ededede geffdZ dZd dZdZdS)!MetricsCollectorzv Collector for Hindsight API metrics. Provides methods to record latency and token usage for operations. ct|_ddlm}|j|_|jddd|_|jddd |_ |jd d d|_ |jd d d|_ |jddd|_ |jddd|_ |jddd|_|jddd|_|jddd|_|d|_dS)N) get_configr+z+Duration of Hindsight operations in secondss)name descriptionunitzhindsight.operation.totalz#Total number of operations executed operationsr/z$Duration of LLM API calls in secondszhindsight.llm.tokens.inputz$Number of input tokens for LLM callstokenszhindsight.llm.tokens.outputz&Number of output tokens from LLM callszhindsight.llm.calls.totalzTotal number of LLM API callscallsr0z$Duration of HTTP requests in secondszhindsight.http.requests.totalzTotal number of HTTP requestsrequestsz#hindsight.http.requests.in_progressz#Number of HTTP requests in progress)r8meterconfigrymetrics_include_bank_id_include_bank_idcreate_histogramoperation_durationcreate_counteroperation_total llm_durationllm_tokens_inputllm_tokens_outputllm_calls_totalhttp_request_durationhttp_requests_totalcreate_up_down_counterhttp_requests_in_progress_setup_process_metrics_db_pool)rOrys r__init__zMetricsCollector.__init__s[[ &&&&&& * D#'*"="=/=jqt#># #  $z88,:_fr 9   !J77)7]dg8   !% 9 9-;ahp!:! ! "&!:!:.]dn$=$ $  *.)J)J6=*K* * & ##%%%04 rrENrFrGrHrIrJc#Ktj}||td}|jr||d<|r||d<|rt||d<d} dVn#t$rd}wxYw tj|z } t||d<|j| ||j d |dS#tj|z } t||d<|j| ||j d |wxYw) aL Context manager to record operation duration and status. Usage: with metrics.record_operation("recall", bank_id="user123", source="api", budget="mid", max_tokens=4096): # ... perform operation pass Args: operation: Operation name (retain, recall, reflect, consolidation) bank_id: Memory bank ID source: Source of the operation (api, reflect, internal) budget: Optional budget level (low, mid, high) max_tokens: Optional max tokens for the operation )rFrHtenantrGrIrJTNFrVrx) timerrrl Exceptionlowerrrecordradd) rOrFrGrHrIrJ start_time attributesrVrSs rrPz!MetricsCollector.record_operation7s0Y[[ "!mm    ,$+Jy !  *#)Jx  7'*:J| $ 4 EEEE   G    y{{Z/H$'LL$6$6$8$8Jy !  # * *8Z @ @ @  $ $Q 3 3 3 3 3y{{Z/H$'LL$6$6$8$8Jy !  # * *8Z @ @ @  $ $Q 3 3 3 3sACA%%CA2E rTr?rQrRrSrTrUrVc|||t|td}|j|||jd||dkr0i|dt|i} |j|| |dkr2i|dt|i} |j || dSdS)rX)r?rQrRrVrrxr token_bucketN) rlrrrrrrr%rr) rOr?rQrRrSrTrUrVbase_attributesinput_attributesoutput_attributess rrZz MetricsCollector.record_llm_callls0!7||))++!mm      ?;;;   O444 !   !  0 > >     ! % %l4D E E E 1  !!! 0 ? ?!!   " & &}6G H H H H H  rr[r\r]c#Ktj}||d}|jd| dVtj|z }|}|dzd}t} i|t ||| d} |j|| |jd| |jd|dS#tj|z }|}|dzd}t} i|t ||| d} |j|| |jd| |jd|wxYw)a  Context manager to record HTTP request metrics. Usage: status_code = [200] # Use list for mutability with metrics.record_http_request("GET", "/api/banks", lambda: status_code[0]): # ... handle request status_code[0] = response.status_code Args: method: HTTP method (GET, POST, etc.) endpoint: Request endpoint path status_code_getter: Callable that returns the status code after request completes )r[r\rxNr#xx) status_code status_classr)rrrrrlrrr) rOr[r\r]rrrSrrrrs rr`z$MetricsCollector.record_http_requests Y[[ %+BB &**1o>>> D EEEy{{Z/H,,..K)S0444L!]]F!";// , J  & - -h C C C  $ ( (J 7 7 7  * . .r? C C C C C'y{{Z/H,,..K)S0444L!]]F!";// , J  & - -h C C C  $ ( (J 7 7 7  * . .r? C C C Cs CB E;c(tdSd}d}d}d}|jd|gdd |jd |gd d |jd |gdd |jd|gdd dS)z-Set up observable gauges for process metrics.Nc3K ttj}tj|jddiVtj|jddiVdS#t$rYdSwxYw)zGet process CPU times.typeusersystemN) _resource_mod getrusage RUSAGE_SELFr Observationru_utimeru_stimer)_optionsrusages r get_cpu_timesz>MetricsCollector._setup_process_metrics..get_cpu_timess &001JKK)&/FF;KLLLLL)&/FH;MNNNNNNN    sA A&& A43A4c3K ttj}|j}t jjdkr|dz}tj|ddiVdS#t$rYdSwxYw)z"Get process memory usage in bytes.Linuxirrss_maxN) rrr ru_maxrssosunamesysnamerrr)rrmax_rsss rget_memory_usagezAMetricsCollector._setup_process_metrics..get_memory_usages &001JKK *8::%00tOG)'FI3FGGGGGGG    sA%A++ A98A9c3^K tjdr9ttjd}t j|VdSttj \}}t j|ddiVdS#t$rYdSwxYw)z$Get number of open file descriptors.z /proc/self/fdlimitsoftN) rpathexistslenlistdirrrr getrlimit RLIMIT_NOFILEr)rcountrhards rget_open_file_descriptorszJMetricsCollector._setup_process_metrics..get_open_file_descriptorss 7>>/22G ? ; ;<.get_thread_countsS ))*@*B*BCCCCCCC    s '- ;;zhindsight.process.cpu.secondszProcess CPU time in secondsrzr{ callbacksr|r}zhindsight.process.memory.byteszProcess memory usage in bytesByzhindsight.process.open_fdszNumber of open file descriptorsz{fds}zhindsight.process.threadszNumber of active threadsz {threads})rrcreate_observable_gauge)rOrrrrs rrz'MetricsCollector._setup_process_metricss  F             **0$o5 +    **1'(7 +    **-019 +    **,'(2 +     rrarbc<||_|dS)z Set the database pool for metrics collection. Args: pool: asyncpg connection pool instance N)r_setup_db_pool_metricsrds rrezMetricsCollector.set_db_pools#  ##%%%%%rc(fd}fd}fd}fd}jd|gddjd |gd djd |gd djd |gdddS)z3Set up observable gauges for database pool metrics.c3KjA tjjVdS#t$rYdSwxYwdS)zGet current pool size.N)rrrget_sizerrrOs r get_pool_sizez>MetricsCollector._setup_db_pool_metrics..get_pool_size'si}(!-dm.D.D.F.FGGGGGGG DD)(-; A A c3KjA tjjVdS#t$rYdSwxYwdS)z'Get number of free connections in pool.N)rrr get_idle_sizerrs rget_pool_free_sizezCMetricsCollector._setup_db_pool_metrics..get_pool_free_size/si}(!-dm.I.I.K.KLLLLLLL DD)(rc3KjA tjjVdS#t$rYdSwxYwdS)zGet pool minimum size.N)rrr get_min_sizerrs rget_pool_min_sizezBMetricsCollector._setup_db_pool_metrics..get_pool_min_size7i}(!-dm.H.H.J.JKKKKKKK DD)(rc3KjA tjjVdS#t$rYdSwxYwdS)zGet pool maximum size.N)rrr get_max_sizerrs rget_pool_max_sizezBMetricsCollector._setup_db_pool_metrics..get_pool_max_size?rrzhindsight.db.pool.sizez)Current number of connections in the poolz {connections}rzhindsight.db.pool.idlez&Number of idle connections in the poolzhindsight.db.pool.minzMinimum pool sizezhindsight.db.pool.maxzMaximum pool sizeN)rr)rOrrrrs` rrz'MetricsCollector._setup_db_pool_metrics$s0                     **)$oC +    **))*@ +    **(()+ +    **(()+ +     rrfrgrh)r9rirjrkrrrlrmrPrnrorZrr`rrerr$rrrvrvs 848484t !!% 24242424 24 d 24 $J 242424^24t2I2I2I2I 2I  2I  2I2I2I2I2I2Ih+D#+D+DRZ[]_b[bRc+D+D+D^+DZJ J J X&&&&> > > > > rrv_metrics_collectorctS)zs Get the global metrics collector instance. Returns a no-op collector if metrics are not initialized. )rr$rrget_metrics_collectorris rc,tatS)zh Create and set the global metrics collector. Should be called after initialize_metrics(). )rvrr$rrcreate_metrics_collectorrrs*++ r)r&r')/rk importlibloggingrutil find_spec import_modulerrr contextlibrtypingrr opentelemetryr!opentelemetry.exporter.prometheusropentelemetry.sdk.metricsr opentelemetry.sdk.metrics.viewr r opentelemetry.sdk.resourcesr asyncpgrlrr4r5r6rmr% getLoggerr9loggerr:r@r8rDrqrvr__annotations__rrr$rrrs    7@~7O7OPZ7[7[e' ' 333ae  %%%%%%********!!!!!!DDDDDD333333SSSSSSSS000000NNN S    h[^"#"#""""J  8 $ $ 99S9S9999x. . . . . . . . b/Dl l l l l +l l l ` ,@+?+A+A(AAA3"2r