$jh dZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl Z ddl mZmZmZmZ ddlZddlmZdZn #e$rdZYnwxYwd d lmZmZmZmZmZd d lmZd d l m!Z!dZ"ej#e$Z%ej#d&ej'e(ej)ee*eZ+ej,e+Z-de*dzde*dzfdZ.Gdde/Z0de*de fdZ1e2hdZ3de*de4fdZ5 d,de*de*de*de*de*de*dzde*dzd e6e*e fdzd!e6e*e*fdzd"e*dzd#e*dzd$e d%e7dzd&e6e*e fdzde fd'Z8Gd(d)Z9Gd*d+Z:e9Z;dS)-z9 LLM wrapper for unified configuration across providers. N)Path)Any)APIConnectionErrorAPIStatusError AsyncOpenAILengthFinishReasonError)service_accountTF)DEFAULT_LLM_MAX_CONCURRENTDEFAULT_LLM_TIMEOUTENV_LLM_GROQ_SERVICE_TIERENV_LLM_MAX_CONCURRENTENV_LLM_TIMEOUT)get_metrics_collector) TokenUsageihttpxtextreturnc>|dS|s|Stjdd|S)a Sanitize text by removing characters that break downstream systems. Removes: - ASCII control characters (0x00-0x08, 0x0B-0x0C, 0x0E-0x1F, 0x7F): break json.loads and PostgreSQL UTF-8 encoding; tab (0x09), newline (0x0A), and carriage return (0x0D) are preserved as they are valid in text and JSON. - Unicode surrogates (U+D800-U+DFFF): Invalid in UTF-8, break LLM APIs Surrogate characters are used in UTF-16 encoding but cannot be encoded in UTF-8. They can appear in Python strings from improperly decoded data (e.g., from JavaScript or broken files). Control characters commonly appear in LLM output embedded inside JSON string values. Nz-[\x00-\x08\x0b\x0c\x0e-\x1f\x7f\ud800-\udfff])resub)rs h/home/rurouni/.hermes/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/llm_wrapper.pysanitize_llm_outputr3s0 |t  6BB M MMceZdZdZdS)OutputTooLongErrora  Bridge exception raised when LLM output exceeds token limits. This wraps provider-specific errors (e.g., OpenAI's LengthFinishReasonError) to allow callers to handle output length issues without depending on provider-specific implementations. N)__name__ __module__ __qualname____doc__rrrrIs Drrrawc|}|dr]d|vr|dddn |dd}|dr |dd}|} t j|S#tj$r-tjdd|}t j|cYSwxYw) u Robustly parse JSON returned by an LLM. Handles common LLM output quirks: 1. Markdown code fences (```json ... ```) — strip them before parsing. 2. Embedded control characters (\x00-\x1f, \x7f) — replace with space and retry if the initial parse fails. Args: raw: Raw text returned by the LLM. Returns: Parsed Python object (dict, list, etc.). Raises: json.JSONDecodeError: If the text cannot be parsed even after cleanup. z``` rNz[\x00-\x1f\x7f] ) strip startswithsplitendswithjsonloadsJSONDecodeErrorrr)r$rcleaneds rparse_llm_jsonr2Us$ 99;;D u)-tzz$""1%%48 ==   9Dzz||#z$  ###&+S$77z'""""" #sB9CC> mocknoneollamabedrocklitellmllamacpplmstudiovertexai litellmrouter claude-code openai-codexproviderc8|tvS)zAReturn True if the given provider requires an API key to operate.)lower_PROVIDERS_WITHOUT_API_KEY)r>s rrequires_api_keyrBs >>  #= ==rapi_keybase_urlmodelreasoning_effortgroq_service_tieropenai_service_tier extra_bodydefault_headersvertexai_project_idvertexai_regionvertexai_credentialsgemini_safety_settingslitellmrouter_configc :ddlm}ddlm}m}m}m}m}m}m }m }m }m }| }|dkr||||||S|dkr||||||S|dkr||||||S|dkr||||||S|d vr||||||| | | | S|d kr||||||| S|d kr||||||S|dkr"| std|||||| |S|dkr,|dr|nd|}||||||S|dkrDddlm}|}|||||||j|j|j|j|j|j S|dvr|||||||||Std|)a Factory function to create the appropriate LLM provider implementation. Args: provider: Provider name ("openai", "groq", "ollama", "gemini", "anthropic", etc.). api_key: API key (may be None for local providers or OAuth providers). base_url: Base URL for the API. model: Model name. reasoning_effort: Reasoning effort level for supported providers. groq_service_tier: Groq service tier (for Groq provider) - "on_demand", "flex", or "auto". openai_service_tier: OpenAI service tier (for OpenAI provider) - None (default) or "flex" (50% cheaper). extra_body: Extra body params merged into OpenAI-compatible API calls. default_headers: Custom headers passed as ``default_headers`` to provider SDK clients (used by operators routing through proxies / request-tracing middleware). Currently wired into the Anthropic provider; other providers may opt in as needed. vertexai_project_id: Vertex AI project ID (for VertexAI provider). vertexai_region: Vertex AI region (for VertexAI provider). vertexai_credentials: Vertex AI credentials object (for VertexAI provider). Returns: LLMInterface implementation for the specified provider. r) LLMInterface) AnthropicLLM ClaudeCodeLLMCodexLLM GeminiLLM LiteLLMLLMLiteLLMRouterLLM LlamaCppLLMMockLLMNoneLLMOpenAICompatibleLLMr=)r>rCrDrErFr<r3r4geminir:) r>rCrDrErFrKrLrMrN anthropic)r>rCrDrErFrJr7r;zProvider 'litellmrouter' requires a config object. Set HINDSIGHT_API_LLM_LITELLMROUTER_CONFIG (or the per-op variant) to a JSON object accepted by litellm.Router. See https://docs.litellm.ai/docs/routing.)r>rCrDrEconfigrFr6zbedrock/r8r  get_config) r>rCrDrErF model_path gpu_layers context_size chat_format no_grammar extra_args) openaigroqr5r9minimaxdeepseekvolcano openrouterzai)r>rCrDrErFrGrHrIzUnknown provider: ) llm_interfacerQ providersrRrSrTrUrVrWrXrYrZr[r@ ValueErrorr+r_rallamacpp_model_pathllamacpp_gpu_layersllamacpp_context_sizellamacpp_chat_formatllamacpp_no_grammarllamacpp_extra_args)r>rCrDrErFrGrHrIrJrKrLrMrNrOrQrRrSrTrUrVrWrXrYrZr[provider_lower bedrock_modelrar_s rcreate_llm_providerrzsL,+++++                        ^^%%N''x-      = ( (}-      6 ! !w-      6 ! !w-      1 1 1y- 3+!5#9     ; & &|-+      9 $ $z-      ? * *# <   '-      9 $ $!&!1!1*!=!=UCUeCUCU z-      : % %''''''{-115311        #"-/ 3!    8h88999rceZdZdZ d9dededededed edzd edzd edzd eeefdzd eeefdzdeeefdzfdZe defdZ e defdZ d:dZ d;deeeefdedzde dzdedzdede ded ed!ed"ed#edefd$Z d LLMProviderzc Unified LLM provider. Supports OpenAI, Groq, Ollama (OpenAI-compatible), and Gemini. lowNr>rCrDrErFrGrHrNrIrJrOc ||_||_||_||_||_| |_||_||_||_ | |_ | |_ |j ,ddl m }  | j|_ n#t$rYnwxYwgd} |j| vr-t!d|jdd| |js|jdkrd |_nq|jd krd |_n^|jd krd |_nK|jdkrd|_n8|jdkrd|_n%|jdkrd|_n|jdkrd|_d}d}d}|jdkrddl m}|}|j}|st!d|jpd}|j}|rUt,st!dt.j|dg}t4d||jdr!|jt;dd|_t4d|d |d!|jd"|rd#nd$|jd%vr5|j .ddl m }  | }|j|_ n#t$rYnwxYw|j}|jd&kr+|)ddl m }  | j}n#t$rd}YnwxYwtA|j|j|j|j|j|j|j|j |j ||||j |'|_!g|_"d|_#dS)(u Initialize LLM provider. Args: provider: Provider name ("openai", "groq", "ollama", "gemini", "anthropic", "lmstudio"). api_key: API key. base_url: Base URL for the API. model: Model name. reasoning_effort: Reasoning effort level for supported providers. groq_service_tier: Groq service tier ("on_demand", "flex", "auto") - from config. openai_service_tier: OpenAI service tier (None or "flex") - from config. gemini_safety_settings: Safety settings for Gemini/VertexAI providers. extra_body: Extra body params merged into OpenAI-compatible API calls. default_headers: Custom headers passed as ``default_headers`` to provider SDK clients. Used by operators routing through proxies / request-tracing middleware. Falls back to ``HindsightConfig.llm_default_headers`` (env: ``HINDSIGHT_API_LLM_DEFAULT_HEADERS``) when ``None``. litellmrouter_config: Provider-specific config for ``provider="litellmrouter"``. JSON object passed verbatim to ``litellm.Router(**config)`` — see https://docs.litellm.ai/docs/routing. Ignored unless ``provider == "litellmrouter"``. When None and the provider is ``litellmrouter``, falls back to ``HindsightConfig.llm_litellmrouter_config``. Nr )_get_raw_config)rhrir5r]r^r9r8r:r=r<r3r4rjrkr7r;r6rlrmrnzInvalid LLM provider: z. Must be one of: z, rizhttps://api.groq.com/openai/v1r5zhttp://localhost:11434/v1r9zhttp://localhost:1234/v1rjzhttps://api.minimax.io/v1rkzhttps://api.deepseek.comrmzhttps://openrouter.ai/api/v1rnz#https://api.z.ai/api/coding/paas/v4r:r`zhHINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID is required for Vertex AI provider. Set it to your GCP project ID.z us-central1zdVertex AI service account auth requires 'google-auth' package. Install with: pip install google-authz.https://www.googleapis.com/auth/cloud-platform)scopesz&Vertex AI: Using service account key: zgoogle/zVertex AI: project=z , region=z, model=z, auth=r ADCr\r;)r>rCrDrErFrGrHrIrJrKrLrMrNrO)$r@r>rCrDrErFrOrGrHrNrIrJr_rllm_default_headers Exceptionrqjoinrallm_vertexai_project_idllm_vertexai_region llm_vertexai_service_account_keyVERTEXAI_AVAILABLEr Credentialsfrom_service_account_fileloggerinfor+lenllm_gemini_safety_settingsllm_litellmrouter_configrz_provider_impl _mock_calls_mock_response)selfr>rCrDrErFrGrHrNrIrJrOrvalid_providersrKrLrMrar_service_account_key raw_config router_configs r__init__zLLMProvider.__init__XsJ!((     0$8!!2#6 &<#$ /   ' 0 0 0 0 0 0 '6'8'8'L$$       , = / /sdmssW[W`W`apWqWqsstt t} F}&& @ (** ; *,, : )++ ; *,, : ,.. > %'' E ## =J & & + + + + + +Z\\F"("@ &  5 %8IMO"("I # \)$@(7'B'\'\'LM(](($ ZEXZZ[[[z$$Y// :!ZI(8(89 KKb&9bbObbbbAT,_,=,=Z_bb    =2 2 2t7R7Z 0 0 0 0 0 0 ,_.. .8.S++    04/H =O + + 0E 0 0 0 0 0 0 % / 1 1 J  % % % $  %2]L]*!2"4 $ 8 0 3+!5#'#>!.   $(*#'s6.B BB9J JJ;K KKrcTddlm}t|j|r |jjSdS)aX Get the OpenAI client for OpenAI-compatible providers. This property provides backward compatibility for code that directly accesses the _client attribute (e.g., benchmarks, memory_engine). Returns: AsyncOpenAI client instance for OpenAI-compatible providers, or None for other providers. r)r[N)providers.openai_compatible_llmr[ isinstancer_client)rr[s rrzLLMProvider._clients> IHHHHH d)+> ? ? /&. .trcTddlm}t|j|r |jjSdS)a3 Get the Gemini client for Gemini/VertexAI providers. This property provides backward compatibility for code that directly accesses the _gemini_client attribute. Returns: genai.Client instance for Gemini/VertexAI providers, or None for other providers. r)rUN)providers.gemini_llmrUrrr)rrUs r_gemini_clientzLLMProvider._gemini_client+s< 433333 d)9 5 5 /&. .trcHK|jd{VdS)z Verify that the LLM provider is configured correctly by making a simple test call. Raises: RuntimeError: If the connection test fails. N)rverify_connectionrs rrzLLMProvider.verify_connection<s5!3355555555555rmemory ?N@Fmessagesresponse_formatmax_completion_tokens temperaturescope max_retriesinitial_backoff max_backoffskip_validation strict_schema return_usagec Kddlm} |dnd} | d|jd|| t4d{V|j||||||||| | |  d{V}|jd kr9d d lm}t|j|r|j |_ |cdddd{VS#1d{VswxYwYdS) a Make an LLM API call with retry logic. Args: messages: List of message dicts with 'role' and 'content'. response_format: Optional Pydantic model for structured output. max_completion_tokens: Maximum tokens in response. temperature: Sampling temperature (0.0-2.0). scope: Scope identifier for tracking. max_retries: Maximum retry attempts. initial_backoff: Initial backoff time in seconds. max_backoff: Maximum backoff time in seconds. skip_validation: Return raw JSON without Pydantic validation. strict_schema: Use strict JSON schema enforcement (OpenAI only). Guarantees all required fields. return_usage: If True, return tuple (result, TokenUsage) instead of just result. Returns: If return_usage=False: Parsed response if response_format is provided, otherwise text content. If return_usage=True: Tuple of (result, TokenUsage) with token counts from the LLM call. Raises: OutputTooLongError: If output exceeds token limits. Exception: Re-raises API errors after retries exhausted. r  set_stageNz +structuredrllm..) rrrrrrrrrrrr3rrY) worker.stagerr>_global_llm_semaphorercallproviders.mock_llmrYrget_mock_callsr)rrrrrrrrrrrrr structuredresultrYs rrzLLMProvider.callEsT -,,,,,&5&A]]r  <<<< <<===(        .33! /&;'' /' /+)4        F }&&777777d17;;L'+':'I'I'K'KD$3                              sA1B88 CCtools>@auto tool_choiceLLMToolCallResultc Kddlm} | d|jd|dt4d{V|j|||||||||  d{V} |jdkr9d d lm} t|j| r|j |_ | cdddd{VS#1d{VswxYwYdS) a Make an LLM API call with tool/function calling support. Args: messages: List of message dicts. Can include tool results with role='tool'. tools: List of tool definitions in OpenAI format. max_completion_tokens: Maximum tokens in response. temperature: Sampling temperature (0.0-2.0). scope: Scope identifier for tracking. max_retries: Maximum retry attempts. initial_backoff: Initial backoff time in seconds. max_backoff: Maximum backoff time in seconds. tool_choice: How to choose tools - "auto", "none", "required", or {"type": "function", "function": {"name": "..."}} Returns: LLMToolCallResult with content and/or tool_calls. r rrrz+toolsN) rrrrrrrrrr3rr) rrr>rrcall_with_toolsrrYrrr) rrrrrrrrrrrrrYs rrzLLMProvider.call_with_toolss: -,,,,, 666666777(        .>>!&;'' /''?        F}&&777777d17;;L'+':'I'I'K'KD$/                              sA/B// B9<B9fnc|jdkr7ddlm}t|j|r|j|dSdSdS)zISet a callback invoked on each call() instead of the fixed mock response.r3rrN)r>rrYrrset_response_callback)rrrYs rrz!LLMProvider.set_response_callbackse =F " " 3 3 3 3 3 3$-w77 >#99"===== # " > >rresponsec||_|jdkr7ddlm}t |j|r|j|dSdSdS)z+Set the response to return from mock calls.r3rrN)rr>rrYrrset_mock_response)rrrYs rrzLLMProvider.set_mock_responsesr' =F " " 3 3 3 3 3 3$-w77 @#55h????? # " @ @rc|jdkr4ddlm}t|j|r|jS|jS)z$Get the list of recorded mock calls.r3rr)r>rrYrrrrrrYs rrzLLMProvider.get_mock_callssV =F " " 3 3 3 3 3 3$-w77 <*99;;;rcg|_|jdkr6ddlm}t |j|r|jdSdSdS)zClear the recorded mock calls.r3rrN)rr>rrYrrclear_mock_callsrs rrzLLMProvider.clear_mock_callssm =F " " 3 3 3 3 3 3$-w77 7#4466666 # " 7 7rctjdz dz }|std|dt |5}t j|}dddn #1swxYwY|d}|dkrtd||d i}|d }|d }|std ||fS) a Load OAuth credentials from ~/.codex/auth.json. Returns: Tuple of (access_token, account_id). Raises: FileNotFoundError: If auth file doesn't exist. ValueError: If auth file is invalid. z.codexz auth.jsonzCodex auth file not found: z> Run 'codex auth login' to authenticate with ChatGPT Plus/Pro.N auth_modechatgptz#Expected auth_mode='chatgpt', got: tokens access_token account_idzGNo access_token found in Codex auth file. Run 'codex auth login' again.) rhomeexistsFileNotFoundErroropenr.loadgetrq)r auth_filefdatarrrrs r_load_codex_authzLLMProvider._load_codex_authsGIKK(*[8 !! #xixxx )__ 9Q< >{?R S S S  ! !"> ? ? H HI\ ] ] ] LLA B B B B B   n  sA>B B! BB!r_ConfiguredLLMProviderc,t||jS)a Return a configured wrapper for a specific bank operation. The wrapper applies per-bank overrides (e.g. Gemini safety settings) to every ``call()`` / ``call_with_tools()`` invocation without changing the underlying provider or its long-lived client connection. Args: config: Resolved ``HindsightConfig`` for the current bank/request. Returns: A ``ConfiguredLLMProvider`` that delegates to this provider with the supplied config applied. )rr)rr_s r with_configzLLMProvider.with_config-s%T6+LMMMrcZK|jr!|jd{VdSdS)z3Clean up resources (e.g. stop llamacpp subprocess).N)rcleanuprs rrzLLMProvider.cleanup>sH   0%--// / / / / / / / / / 0 0rc ddlm}m}m}m}m}m}m}m}tj ||} tj |d} | st| sn| st|dtj |d} tj ||} tjtj |d} tjtj |d}|| | | | d| |S)zECreate provider from environment variables using config.py constants.r )DEFAULT_LLM_MODELDEFAULT_LLM_PROVIDERENV_LLM_API_KEYENV_LLM_BASE_URLENV_LLM_DEFAULT_HEADERSENV_LLM_EXTRA_BODY ENV_LLM_MODELENV_LLM_PROVIDERrzV environment variable is required (unless using openai-codex, claude-code, or litellm)nullr})r>rCrDrErFrIrJ)r_rrrrrrrrosgetenvrBrqr.r/)clsrrrrrrrrr>rCrDrErIrJs rfrom_envzLLMProvider.from_envCsY 9-/CDD)OR00 /99   "zzz 9-r22 -):;;Z *)8 8888t>>>>>>@#@$@@@@ T     7777!(%S/!(!(!(!(F6N#N*ANNNN"0000 $ $ $ [$ $ $ rr|ceZdZdZdddedzddfdZdedefd Zd ee eefd edefd Z d ee eefd ee eefd eddfdZ dS)ruH Thin wrapper around LLMProvider that applies bank-specific config to every call. Obtained via ``LLMProvider.with_config(resolved_config)``. The wrapper sets any provider-specific overrides (currently Gemini safety settings) immediately before each call using a ContextVar token, then resets it afterwards — so nesting is safe and the configuration cannot leak across operations. All attribute access falls through to the underlying provider so callers that read ``llm.provider``, ``llm.model``, etc. continue to work without any changes. r>r|rNNrcvt|d|t|d|dS)N _provider_gemini_safety_settings)object __setattr__)rr>rNs rrzConfiguredLLMProvider.__init__zs:4h7774!:>>>>>$(()@)@G`)a)abb .00{CCHeeRZe^deeeeeeee e & &u - - - - & &u - - - -s ,A;;Brrc,Kddlm}|t|d} t|djd||d|d{V ||S#||wxYw)Nrrr r )rrr#)rrrr rrr)rrrrrrs rrz%ConfiguredLLMProvider.call_with_toolss ?>>>>>$(()@)@G`)a)abb .00{CCS!28  ! & &u - - - - & &u - - - -s -A<<B) rr r!r"rrrrrrrrr#rrrrks  TTt TX\TTTTIIIIII .4S#X#7.3.3.....tCH~&.DcN#. .  ......rr) NNNNNNNNN)r+s   [[[[[[[[[[[[------,+++++''''''  8 $ $'##GO444c)")$:CC@Z<[<[\\]]))*=>>NcDjNS4ZNNNN,          !#!#!#!#!#!#H'Y   ">s>t>>>>%)&*(,-1&*"& $*.26::: ::  :  : Tz :t:S#X%:#s(^d*:t:4Z::!4K:sCx.4/: ::::DW W W W W W W W t2.2.2.2.2.2.2.2.l  s A AA