$j QdZddlZddlZddlmZmZddlmZmZddlm Z m Z ej e Z Gdde ZGdd e ZGd d eZGd d eZGddeZdS)z Query analysis abstraction for the memory system. Provides an interface for analyzing natural language queries to extract structured information like temporal constraints. N)ABCabstractmethod)datetime timedelta) BaseModelFieldcdeZdZUdZedZeed<edZeed<de fdZ d S) TemporalConstraintzl Temporal constraint extracted from a query. Represents a time range with start and end dates. z#Start of the time range (inclusive)) description start_datez!End of the time range (inclusive)end_datereturncp|jdd|jdS)N%Y-%m-%d to )r strftimer selfs k/home/rurouni/.hermes/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/query_analyzer.py__str__zTemporalConstraint.__str__s7/**:66``DMeZdZUdZeddZedzed<dS) QueryAnalysisz Result of analyzing a natural language query. Contains extracted structured information like temporal constraints. Nz%Extracted temporal constraint, if any)defaultr temporal_constraint)rrrrrr"r rrrrr r sV 6;U"I666+d2rr cVeZdZdZeddZed dededzdefdZ dS) QueryAnalyzerz Abstract base class for query analysis. Implementations analyze natural language queries to extract structured information like temporal constraints, entities, etc. rNcdS)z Load the query analyzer model. This should be called during initialization to load the model and avoid cold start latency on first analyze() call. Nrrs rloadzQueryAnalyzer.load4s  rqueryreference_datecdS)a Analyze a natural language query. Args: query: Natural language query to analyze reference_date: Reference date for relative terms (defaults to now) Returns: QueryAnalysis containing extracted information Nr)rr'r(s ranalyzezQueryAnalyzer.analyze>s  rrNN) rrrrrr&rrr r*rrrr$r$,s|   ^    S  (T/  ]    ^    rr$cVeZdZdZdZd dZd dededzdefdZ dedede dzfd Z dS) DateparserQueryAnalyzeraF Query analyzer using dateparser library. Uses dateparser to extract temporal expressions from natural language queries. Supports 200+ languages including English, Spanish, Italian, French, German, etc. Performance: - ~10-50ms per query - No model loading required (lazy import on first use) cd|_dS)z%Initialize dateparser query analyzer.N) _search_datesrs r__init__z DateparserQueryAnalyzer.__init__Zs!rrNc\|j$ddlm}||_|ddSdS)zLoad dateparser and warm up internal data structures. Triggers the real initialization cost (regex tables, timezone data) at load time so the first actual recall doesn't pay the cold-start penalty. Nr) search_datestoday)r0dateparser.searchr3)rr3s rr&zDateparserQueryAnalyzer.load^sI   % 6 6 6 6 6 6!-D    w ' ' ' ' ' & %rr'r(c |tj}|}|||}|t |S||ddd} |||}nU#t$rH}t dt|j |t dcYd}~Sd}~wwxYw|st dShd fd |D}|st dS|d \} } | d d d d } | d d d d } t t| | S)a Analyze query using dateparser. Extracts temporal expressions from the query text. Supports multiple languages automatically. Args: query: Natural language query (any language) reference_date: Reference date for relative terms (defaults to now) Returns: QueryAnalysis with temporal_constraint if found Nr"pastF) RELATIVE_BASEPREFER_DATES_FROMRETURN_AS_TIMEZONE_AWARE)settingszFdateparser raised %s on query (treating as no temporal constraint): %s> docanfrimaymonsatsunthutuewedwillmarchcpg|]2\}}|vst|dk.||f3S))lowerlen).0textdatefalse_positivess r z3DateparserQueryAnalyzer.analyze..sIxxx*$4::<.constraintI% ==a!QR=SS"RPVWW rz&\b(yesterday|ayer|ieri|hier|gestern)\bdaysz(\b(today|hoy|oggi|aujourd\'?hui|heute)\bz'\b(a\s+)?couple\s+(of\s+)?days?\s+ago\brJz\b(a\s+)?few\s+days?\s+ago\bz(\b(a\s+)?couple\s+(of\s+)?weeks?\s+ago\b)weeksz\b(a\s+)?few\s+weeks?\s+ago\bz)\b(a\s+)?couple\s+(of\s+)?months?\s+ago\bZz\b(a\s+)?few\s+months?\s+ago\b<ui\b(last\s+week|la\s+semana\s+pasada|la\s+settimana\s+scorsa|la\s+semaine\s+derni[eè]re|letzte\s+woche)\bz\\b(last\s+month|el\s+mes\s+pasado|il\s+mese\s+scorso|le\s+mois\s+dernier|letzten?\s+monat)\bdayud\b(last\s+year|el\s+a[ñn]o\s+pasado|l\'anno\s+scorso|l\'ann[ée]e\s+derni[eè]re|letztes?\s+jahr)\b z\b(last\s+weekend|el\s+fin\s+de\s+semana\s+pasado|lo\s+scorso\s+fine\s+settimana|le\s+week-?end\s+dernier|letztes?\s+wochenende)\br ) z$january|enero|gennaio|janvier|januaru-february|febrero|febbraio|f[ée]vrier|februarumarch|marzo|mars|m[äa]rzzapril|abril|aprile|avrilzmay|mayo|maggio|maizjune|junio|giugno|juin|junizjuly|julio|luglio|juillet|juliuaugust|agosto|ao[uû]tz(september|septiembre|settembre|septembrez'october|octubre|ottobre|octobre|oktoberznovember|noviembre|novembreu1december|diciembre|dicembre|d[ée]cembre|dezemberz\b(z )\s+(\d{4})\bN) rr research IGNORECASErweekdayrayearitemsintgroup)rr'r(rpdrjfirstrkrdays_since_satrBmonth_patternspattern month_nummatchs rr\z'DateparserQueryAnalyzer._extract_periodsA h X :L     9>r} U U $!2!2!22A:a## # 9@% W W >:nn== = 9? V V f:nya/@/@/@@.S\bcSdSdSdBdee e 94eR] K K f:nya/@/@/@@.S\bcSdSdSdBdee e 9@% W W h:nyq/A/A/AA>T]deTfTfTfCfgg g 95ubm L L h:nyq/A/A/AA>T]deTfTfTfCfgg g 9A5"- X X h:nyb/A/A/AA>T]ceTfTfTfCfgg g 96r} M M i:nyc/B/B/BBNU^dfUgUgUgDghh h 9 x  M   @ #YN4J4J4L4Lq4P%Q%Q%QQE:eUYA->->->%>?? ? 9 k  M   * #**q*11E)++++CKKAK&&E:eS)) ) 9 s  M   L "&*D:htQ22HT2r4J4JKK K 9 R  M  < -4466:a?N""!" 9.#A#A#AAC:c3):):):#:;; ;56=>)*()#$+,./&'8979+-AC   #1"6"6"8"8 . . GYI=W===ubmTTE .5;;q>>** y!44??"4R00CC"4Q::YA=N=N=NNC!z%----- .trr+r,) rrrrr1r&rrr r*r r\rrrr.r.Ms  """ ( ( ( (BoBoSBo(T/Bo]BoBoBoBoHwSw(wGY\`G`wwwwwwrr.ceZdZdZddedefdZdd Zd Zd ed ede dzfd Z dd ed edzde fdZ dZ ded ede dzfdZdS)TransformerQueryAnalyzera# Query analyzer using T5-based generative models. Uses T5 to convert natural language temporal expressions into structured date ranges without pattern matching or regex. Performance: - ~30-80ms on CPU, ~5-15ms on GPU - Model size: ~80M params (~300MB download) google/flan-t5-smallcpu model_namedevicec>||_||_d|_d|_dS)a^ Initialize T5 query analyzer. Args: model_name: Name of the HuggingFace T5 model to use. Default: google/flan-t5-small (~80M params, ~300MB download) Alternative: google/flan-t5-base (~1GB, more accurate) device: Device to run model on ("cpu" or "cuda") N)rr_model _tokenizer)rrrs rr1z!TransformerQueryAnalyzer.__init__5s$%  rrNc|jdS ddlm}m}n#t$rt dwxYwt d|jd||j|_ ||j|_|j |j |j t ddS)z*Load the T5 model for temporal extraction.Nr)AutoModelForSeq2SeqLM AutoTokenizerz`transformers is required for TransformerQueryAnalyzer. Install it with: pip install transformerszLoading query analyzer model: z...zQuery analyzer model loaded) r transformersrr ImportErrorr^inforfrom_pretrainedrtoreval)rrrs rr&zTransformerQueryAnalyzer.loadDs ; " F  I I I I I I I I I   r    IT_IIIJJJ'77HH+;;DOLL  t{###  122222s.c.|dS)z>Lazy load the T5 model for temporal extraction (calls load()).N)r&rs r _load_modelz$TransformerQueryAnalyzer._load_modelWs rr'r(c xddl}|}dtdtffd }dtdtdtfd}|jd |rt d z }|||S|jd |rEt d z z }|||t d zS|jd|rKd } | t d z } | d }||| S|jd|r4j d z } |t| d d t| ddS|jd|r(|d} || | t d zSdd dddddd} | D]3\}}|jd|d|r||}|||cS4d dddddd dddddd }| D]\}}|jd|d!|}|rt| d }|dkrd}n*t||d zd t d z j }|t||d t|||cSdS)"z Extract temporal expressions using rule-based patterns. Handles common patterns reliably and fast. Returns None for patterns that need model-based extraction. rNrrcn|z dz}|dkrd}t|z SNr|rrsrrrdays_agor(s rget_last_weekdayzFTransformerQueryAnalyzer._extract_with_rules..get_last_weekdayfB&..007:a?H1}}!I8$<$<$<< .constraintlrqrz \byesterday\brrrsz\blast\s+week\br|r}z\blast\s+month\br~z\blast\s+year\brrz\blast\s+weekend\brurvrJr)mondaytuesday wednesdaythursdayfridaysaturdaysundayz \blast\s+z\brrrr) januaryfebruaryrHaprilr@junejulyaugust septemberoctobernovemberdecemberz \s+(\d{4})\b) rrKrrr rrrrarrrr)rr'r(rrbrrprrjrrkyrBweekdaysnamenummonthsrrlast_days ` r_extract_with_rulesz,TransformerQueryAnalyzer._extract_with_rules[s  kkmm  =c =h = = = = = =  h X :L     29%{ 3 3 $!2!2!22A:a## # 29' 5 5 @"YN4J4J4L4Lq4P%Q%Q%QQE:eUYA->->->%>?? ? 29(+ 6 6 *"**q*11E)++++CKKAK&&E:eS)) ) 29' 5 5 F#a'A:hq!Q//!R1D1DEE E 29*K 8 8 <""1%%C:c3):):):#:;; ; AA1XYghtuvv!)) ( (ID#ry.d... << ($$S))!z!Q''''' (      Y YID#BI8D888+FFE Y5;;q>>**"99!HH (sQw : :YA=N=N=N NSH!z(4a"8"8(4h:W:WXXXXX  Ytrc tj|}|t|Sdt dtffd }t dz }|d}d d d |d d |d d |d d |d d|d }|ddd}fd| D} 5j j di|ddddd} dddn #1swxYwYj | dd}|} t| S)a Analyze query for temporal expressions. Uses rule-based extraction for common patterns (fast & reliable), falls back to T5 model for complex/unusual patterns. Args: query: Natural language query reference_date: Reference date for relative terms (defaults to now) Returns: QueryAnalysis with temporal_constraint if found Nr7rrcn|z dz}|dkrd}t|z Srrrs rrz:TransformerQueryAnalyzer.analyze..get_last_weekdayrrrrrsruz Today is rzR. Extract date range or "none". June 2024 = 2024-06-01 to 2024-06-30 yesterday = rz last Saturday = z what is the weather = none z =ptiT)return_tensors max_length truncationcLi|] \}}||j!Sr)rr)rMkvrs r z4TransformerQueryAnalyzer.analyze..s-BBB41a!QTT$+&&BBBrryrJFg?)max_new_tokens num_beams do_sample temperaturer)skip_special_tokensr)rr[rr rrrrrr_no_gradrgeneratedecodestrip_parse_generated_output) rr'r(resultr yesterday last_saturdaypromptinputsoutputstemporals ` ` rr*z TransformerQueryAnalyzer.analyzes  !%\^^N))%@@   V<<< <  =c =h = = = = = = #YA%6%6%66 ((++  ~66zBB     + +  2;1C1CJ1O1O  '' 33   :G9O9OPZ9[9[       Y]^^BBBB6<<>>BBB ]]__ w w*dk*vvVvBRS_druvvvvG w w w w w w w w w w w w w w w'' 'MMSSUU//GG::::sE--E14E1ct ddl}|S#t$rddlm}|cYSwxYw)z"Get torch.no_grad context manager.rN) nullcontext)torchno_gradr contextlibr)rrrs rrz!TransformerQueryAnalyzer._no_grads] ! LLL==?? " ! ! ! . . . . . .;==  !s 77rcN|r(|dvrdS ddl}d}|j|||j}|r|d}|d}t j|d}t j|d} |dddd}| d d d d } | |kr"t d |d | dSt|| Sn#ttf$rYdSwxYwdS)a9 Parse T5 generated output into TemporalConstraint. Expected format: "YYYY-MM-DD to YYYY-MM-DD" Args: result: Generated text from T5 reference_date: Reference date for validation Returns: TemporalConstraint if valid output, else None )nonenullnoNrz.(\d{4}-\d{2}-\d{2})\s+to\s+(\d{4}-\d{2}-\d{2})rrrvrrRrWrXrYzInvalid date range: rrZ)rKrrrrrrstrptimerar^r_r ValueErrorAttributeError) rrr(rrr start_strend_strr r s rrz0TransformerQueryAnalyzer._parse_generated_outputs_ --//3III4  IIIGGBIgvr}==E T!KKNN ++a..%.y*EE #,WjAA(//QqXY/ZZ #++BrW]+^^j((NN#T*#T#T(#T#TUUU4)Z(SSSS! T$N+   44 tsC D ;D D"!D")rrr+r,)rrrrrr1r&rrr rr r*rrrrrrr)s    3     3333&RRhRK]`dKdRRRRh7;7;S7;(T/7;]7;7;7;7;r ! ! !,c,8,PbeiPi,,,,,,rr)rloggingrabcrrrrpydanticrr getLoggerrr^r r r$r.rrrrrsu ########((((((((%%%%%%%%  8 $ $ a a a a a a a a     I        C   BYYYYYmYYYxvvvvv}vvvvvr