Jj UdZddlZddlZddlZddlZddlZddlmZmZm Z m Z m Z ddl Z e rddl mZddlmZmZmZmZmZmZddlmZmZmZddlmZmZddlmZdZe e ed <dZ e e ed <dZ!e e ed <dZ"e e ed <dZ#e e ed <ddl$m%Z%ddl&m'Z'm(Z)m*Z+m,Z,ddl-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3m4Z4ddl5Z5ej6e7Z8de9de9fdZ:de9de;fdZe?hdZ@de9fdZAde9fdZBdZCde9fdZDde9fdZEde9fdZFde9de9fd ZGde9de;fd!ZHde;fd"ZIdeJe9fd#ZKd$ZLd%ZMe%d&d'(ZNdeOfd)ZPd*e9de9fd+ZQd,e9d-e9de e9fd.ZRd-e9d,e9d/eOdeSe9e;ffd0ZTdd1ZUdd3e9d4eOde9fd5ZV dd6ee9d7e9d/e eOde9fd8ZWde;fd9ZXe7d:kr eYd;eYd<eXZZeZ[e;ej\d=d>]Z^e;ej\d?d>]Z_eZreDZ`eYd@e`e`dAkr eYdBne`dCkr eYdDne`dEkr eYdFne`dGkreYdHe:dIne`dJkr eYdKne`dLkr eYdMne_rBeYdNej\d?]adOnTe^r eYdPnFe[reYdQen-eYdRn!eYdSeYdTeeZse5jbdUeYdVeYdWePdXeNjcr0eYdYeNjdeYdZeNjed[eNjdd\n eYd]eYd^eYd_eYd`eYd>eYdaeYdbeYd>eYdceYddeYdeeYdfeYdgeYdheYdieYdjeYdkddllfmgZgmhZhdmdndodpdqdrdsdtdUdud2dvdwd3gdxdyZidzd{dod|d}dpid~d2ddsddddd6gdxdyZjegjkdmdeideXeKddegjkdzdejdeXeKddd dS)ae Standalone Web Tools Module This module provides generic web tools that work with multiple backend providers. Backend is selected during ``hermes tools`` setup (web.backend in config.yaml). When available, Hermes can route Firecrawl calls through a Nous-hosted tool-gateway for Nous Subscribers only. Available tools: - web_search_tool: Search the web for information - web_extract_tool: Extract content from specific web pages Backend compatibility: - Exa: https://exa.ai (search, extract) - Firecrawl: https://docs.firecrawl.dev/introduction (search, extract; direct or derived firecrawl-gateway. for Nous Subscribers) - Parallel: https://docs.parallel.ai (search, extract) - Tavily: https://tavily.com (search, extract) LLM Processing: - Uses OpenRouter API with Gemini 3 Flash Preview for intelligent content extraction - Extracts key excerpts and creates markdown summaries to reduce token usage Debug Mode: - Set WEB_TOOLS_DEBUG=true to enable detailed logging - Creates web_tools_debug_UUID.json in ./logs directory - Captures all tool calls, results, and compression metrics Usage: from web_tools import web_search_tool, web_extract_tool # Search the web results = web_search_tool("Python machine learning libraries", limit=3) # Extract content from URLs content = web_extract_tool(["https://example.com"], format="markdown") N)ListDictAnyOptional TYPE_CHECKING) Firecrawl)r_firecrawl_backend_help_suffix_get_firecrawl_client_get_firecrawl_gateway_url_is_tool_gateway_readycheck_firecrawl_api_key)_normalize_tavily_documents _normalize_tavily_search_results_tavily_request)_get_async_parallel_client_get_parallel_client)_get_exa_client_firecrawl_client_firecrawl_client_config_parallel_client_async_parallel_client _exa_client) DebugSession)build_vendor_gateway_urlpeek_nous_access_tokenread_nous_access_tokenresolve_managed_tool_gateway)managed_nous_tools_enabled%nous_tool_gateway_unavailable_messageprefers_gateway)async_is_safe_urlnormalize_url_for_requestsensitive_query_param_namenamereturnc ddlm}||}n#t$rd}YnwxYw|tj|d}|pdS)uResolve ``name`` via Hermes config-aware env, falling back to process env. Mirrors the SearXNG provider's ``_searxng_url()`` so that values set through Hermes' config/.env layer (``hermes config set``, ``hermes tools``) are honored here too — not just raw process-env exports. Without this, a config-only ``SEARXNG_URL`` (or any provider key) leaves the backend auto-detect cascade and ``check_web_api_key()`` blind to it. See #34290. r) get_env_valueN)hermes_cli.configr' Exceptionosgetenvstrip)r$r'vals 5/home/rurouni/.hermes/hermes-agent/tools/web_tools.py _env_valuer0lsz333333mD!!  {ib!! I2    s  ##c:tt|SN)boolr0)r$s r/_has_envr4s  4  ! !!c~ ddlm}|diS#ttf$ricYSwxYw)z5Load the ``web:`` section from ~/.hermes/config.yaml.r load_configweb)r)r8get ImportErrorr*r7s r/_load_web_configr<s^111111{}}  +++  # s #&<<>exaxaiddgstavilysearxngparallel firecrawl brave-freebackendc|sdS ddlm}||S#t$r'}td||Yd}~dSd}~wwxYw)alReturn a plugin-registered web provider by name, or ``None``. Consults ``agent.web_search_registry`` so backends contributed by the plugin system (which are absent from :data:`_LEGACY_WEB_BACKENDS`) are discoverable during availability/selection resolution. Returns ``None`` on any lookup failure so callers can fall through to legacy checks. Nr) get_providerz.web provider registry lookup failed for %r: %s)agent.web_search_registryrGr*loggerdebug)rErGexcs r/_registered_web_providerrLs| t::::::|G$$$  EwPSTTTttttts AAAct|}|dS t|S#t$r'}td||Yd}~dSd}~wwxYw)u+Availability of a *registered* web provider, or ``None`` if unregistered. Returns ``True``/``False`` when *backend* names a registered provider (calling its ``is_available()``), or ``None`` when it isn't registered — letting the caller fall through to the legacy built-in probes. N)web provider %r.is_available() raised: %sF)rLr3 is_availabler*rIrJ)rEproviderrKs r/"_registered_web_provider_availablerQs~(00HtH))++,,,  @'3OOOuuuuus 6 A'A""A'c ddlm}|S#t$r'}td|gcYd}~Sd}~wwxYw)zCReturn all plugin-registered web providers (empty list on failure).r)list_providersz%web provider registry list failed: %sN)rHrSr*rIrJ)rSrKs r/_list_registered_web_providersrTsj<<<<<<~  AAc tdpd}|tvst ||Sdt dfdt dfdt d fd t d pt d fd tfd t dfdt dfdtff}|D] \}}|r|cS tD]g}|j tvr | r |j cS0#t$r+}td|j |Yd}~`d}~wwxYwd S)zDetermine which web backend to use (shared fallback). Reads ``web.backend`` from config.yaml (set by ``hermes tools``). Falls back to whichever API key is present for users who configured keys manually without running setup. rEr(Nr@TAVILY_API_KEYr= EXA_API_KEYrBPARALLEL_API_KEYrCFIRECRAWL_API_KEYFIRECRAWL_API_URLrA SEARXNG_URLrDBRAVE_SEARCH_API_KEYr?rN)r<r:lowerr-_LEGACY_WEB_BACKENDSrLr4r _ddgs_package_importablerTr$rOr*rIrJ) configuredbackend_candidatesrE availablerPrKs r/ _get_backendrcs#$$((339r@@BBHHJJJ)))-Ej-Q-Q-] 8,--. ''( X0112 h233Tx@S7T7TU ,../ H]++, x 6778 )++, 1  NNN 344ZZ =0 0 0  Z$$&& %}$$$ % Z Z Z LLDhmUX Y Y Y Y Y Y Y Y Z ;sD== E2!E--E2c tdS)ujDetermine which backend to use for web_search specifically. Selection priority: 1. ``web.search_backend`` (per-capability override) 2. ``web.backend`` (shared fallback — existing behavior) 3. Auto-detect from env vars This enables using different providers for search vs extract (e.g. SearXNG for search + Firecrawl for extract). search_get_capability_backendr5r/_get_search_backendris #8 , ,,r5c tdS)uDetermine which backend to use for web_extract specifically. Selection priority: 1. ``web.extract_backend`` (per-capability override) 2. ``web.backend`` (shared fallback — existing behavior) 3. Auto-detect from env vars extractrfrhr5r/_get_extract_backendrl s #9 - --r5 capabilityct}||dpd}|rt |r|St S)zShared helper for per-capability backend selection. Reads ``web.{capability}_backend`` from config; if set and available, uses it. Otherwise falls through to the shared ``_get_backend()``. _backendr()r<r:r]r-_is_backend_availablerc)rmcfgspecifics r/rgrgsm   C:///006B==??EEGGH)(33 >>r5c|pd}|tvrt|}||S|dkrt dS|dkrt dS|dkrt S|dkrt d S|d krt d S|d krt d S|dkrt S|dkr" ddlm}|S#t$rYdSwxYwdS)u4Return True when the selected backend is currently usable. For plugin-registered backends (any name outside :data:`_LEGACY_WEB_BACKENDS`), availability is delegated to the provider's ``is_available()`` via the web_search_registry. This is the single chokepoint through which ``_get_backend``, ``_get_capability_backend``, and ``check_web_api_key`` all resolve availability — fixing custom-provider discovery for every caller at once (issues #28651, #31873, #32698). Built-in backends keep their cheap hardcoded probes below. r(Nr=rWrBrXrCr@rVrAr[rDr\r?r>r)has_xai_credentialsF) r]r-r^rQr4r r_tools.xai_httprtr*)rE registeredrts r/rprp$sP}"##%%++--G***7@@  ! % &&&**++++&((((()))) &&&,.///&')))%   : : : : : :&&(( (   55  5sC-- C;:C;c2 ddl}dS#t$rYdSwxYw)aJReturn True when the ``ddgs`` Python package can be imported. ddgs is the only backend whose availability is driven by a package presence rather than an env var / config entry. Wrapped in a helper so auto-detect and ``_is_backend_available`` share the same check (and tests can monkeypatch a single symbol). rNTF)r?r;)r?s r/r_r_Ps7 t uus  c gdS)uReturn tool metadata env vars for the currently enabled web backends. The gateway env vars are always reported — they're metadata strings used by the tool registry to light up the tool when the variable is set. Gating them on ``managed_nous_tools_enabled()`` only saved string noise in the metadata list, but cost a synchronous HTTP refresh against the Nous portal on every CLI startup (invoked at tool-registration time). The behavioral contract is: if the env var is set, the tool sees it; if not, it doesn't. Not-logged-in users simply don't have the vars set, so the extra entries are harmless. ) rWrXrVrYrZFIRECRAWL_GATEWAY_URLTOOL_GATEWAY_DOMAINTOOL_GATEWAY_SCHEMETOOL_GATEWAY_USER_TOKENrhrhr5r/_web_requires_envr}hs    r5i:i web_toolsWEB_TOOLS_DEBUG)env_varc td}|-t|}tdt |dSn#t t f$rYnwxYwtS)zFResolve the per-page char budget from config, clamped to a sane range.extract_char_limitN )r<r:intmaxmin TypeError ValueErrorDEFAULT_EXTRACT_CHAR_LIMIT)r`values r/_get_extract_char_limitrs %''++,@AA  ! OOEtS0011 1 " z "     %%sAAA'&A'textcdddtfd}tjd}|||}tjdd|}tjdd|}|S) aReplace inline base64 image blobs with labeled markdown links. base64 image payloads are token bombs (a single inline PNG can be tens of thousands of characters), so we never send the raw bytes to the model. But we preserve the fact that an image was there, and its alt text, as an inspectable placeholder. Real (http/https) markdown image links are left untouched so the agent can ``web_extract`` / ``vision_analyze`` them. Transformations: ``![alt](data:image/png;base64,AAAA...)`` -> ``[IMAGE: alt](base64 image omitted)`` ``(data:image/png;base64,AAAA...)`` -> ``[IMAGE]`` bare ``data:image/...;base64,AAAA...`` -> ``[IMAGE]`` mz re.Match[str]r%ch|dpd}|rd|dndS)Naltr(z[IMAGE: ][IMAGE])groupr-)rrs r/_md_replz0convert_base64_images_to_links.._md_repls>wwu~~#**,,$'6 #    Y6r5zD!\[(?P[^\]]*)\]\(\s*data:image/[^;]+;base64,[A-Za-z0-9+/=\s]+\)z0\(\s*data:image/[^;]+;base64,[A-Za-z0-9+/=\s]+\)rz'data:image/[^;]+;base64,[A-Za-z0-9+/=]+)strrecompilesub)rrmd_b64outs r/convert_base64_images_to_linksrs7O77777ZOF **Xt $ $C &DiQT U UC &;Y L LC Jr5urlcontentc ddl}ddlm}ddlm}|dd}|dd||jpd d d }tj d d |dd d pd }| | d dd}||d |dz } t|tkr-|dtdtddt|ddz}| |dt#| S#t$$r'} t&d|| Yd} ~ dSd} ~ wwxYw)aWrite the full extracted page to cache/web and return its absolute path. The file is mounted read-only into remote backends (Docker/Modal/SSH) via credential_files._CACHE_DIRS, so the agent's terminal/read_file tools can page through the complete text on any backend. Returns None on failure (storage is best-effort; truncated content is still returned to the model). rN)urlparse)get_hermes_dirz cache/web web_cacheT)parentsexist_okpage:_z[^A-Za-z0-9._-]-<zutf-8 z.mdz [... stored copy truncated at ,z chars of z2; re-extract a more specific URL for the rest ...])encodingz0Failed to store full web_extract text for %s: %s)hashlib urllib.parserhermes_constantsrmkdirhostnamereplacerrr-sha256encode hexdigestlenMAX_STORED_TEXT_CHARS write_textrr*rIrJ) rrrrr cache_dirhostslugdigestpathrKs r/_store_full_textrs))))))333333"N; << t444 &0&99#sCCv(#t44SbS9??DDN 7 3 344>>@@"Ed00V0000 w<</ / /.../Y7LWYY'llXYYYY  '2224yy  GcRRRtttttsEE E7E22E7 char_limitc Vt||kr|dfSt|dz}||z }|d|}|| d}|d}||dzkr |d|}|d}d|cxkr |dzkrnn ||dzd}t|}t ||} t|t|z} dd d t|d d t|d d |d dg} | rM|ddz} | d| | d| d| dn| d| d|dz|z} | dd| zz } | dfS)uReturn (model_text, was_truncated) for one page's clean content. Pages at or under ``char_limit`` are returned whole. Larger pages get a head+tail window (~75% head / ~25% tail) cut on a markdown line boundary where possible, plus an explicit footer telling the model exactly how much it is seeing, where the full text is stored, and which read_file call pages in the omitted middle. Deterministic — no model involvement. Fg?N g?rr(u=──────── [TRUNCATED] ────────zShowing rz chars (head) + z chars (tail) of z total clean characters.zFull text saved to: z,To read the omitted middle: read_file path="z " offset=zS limit=200 (the file is the complete page; raise/lower offset to page through it).zwFull text could not be stored; re-run web_extract on a more specific URL or use browser_navigate for the complete page.uW─────────────────────────────u+ [... middle omitted — see footer ...] T)rrrfindfindrcountappendjoin)rrr head_budget tail_budgetheadtailnltotal stored_pathshown footer_linesmiddle_start_line model_texts r/_truncate_with_footerrs= 7||z!!~j4'((K{*K /__init__.py`` during plugin discovery. Tool dispatch can be reached from contexts that haven't already triggered discovery — subprocess agent runs, delegate children, standalone scripts, certain test paths — and without it the registry is empty and ``get_provider('firecrawl')`` returns ``None`` even when the user has ``web.extract_backend: firecrawl`` configured and ``FIRECRAWL_API_KEY`` set. The symptom is a misleading "No web extract provider configured" error (issue #27580). Mirrors :func:`tools.browser_tool._ensure_browser_plugins_loaded` exactly: the underlying discovery call is idempotent and cheap on subsequent invocations. r)_ensure_plugins_discoveredz+Web plugin discovery failed (non-fatal): %sN)hermes_cli.pluginsrr*rIwarning)rrKs r/_ensure_web_plugins_loadedr;s{" KAAAAAA""$$$$$ KKK DcJJJJJJJJJ Ks A?Aquerylimitc\ t|}n#ttf$rd}YnwxYwtt |dd}||dddddd} ddlm}|rtd d Stdd l m }m }t}|r ||nd}|| s |}|d d d}n8td|j|||||}t%|didg|d<t)j|dd } t%| |d<t,d|t,| S#t2$r} dt5| } td| | |d<t,d|t,t| cYd} ~ Sd} ~ wwxYw)aE Search the web for information using available search API backend. This function provides a generic interface for web search that can work with multiple backends (Parallel or Firecrawl). Note: This function returns search result metadata only (URLs, titles, descriptions). Use web_extract_tool to get full content from specific URLs. Args: query (str): The search query to look up limit (int): Maximum number of results to return (default: 5) Returns: str: JSON string containing search results with the following structure: { "success": bool, "data": { "web": [ { "title": str, "url": str, "description": str, "position": int }, ... ] } } Raises: Exception: If search fails or API key is not set rrdrrNr) parameterserror results_countoriginal_response_sizefinal_response_size)is_interrupted InterruptedF)success)get_active_search_providerrGzDNo web search provider configured. Run `hermes tools` to set one up.rrz#Web search via %s: '%s' (limit: %d)datar9rrindent ensure_asciirweb_search_toolzError searching web: %sr)rrrrrtools.interruptr tool_errorrrHrrGrisupports_searchrIinfor$rerr:jsondumps_debuglog_callsaver*rrJ) rrdebug_call_datarr_wsp_get_providerrErP response_data result_jsone error_msgs r/rrXsDE  z " E1 s # #E  "#   O5%222222 >   <mU;;; ; #$$$        &''18B$$W---d  8#;#;#=#= 2133H   8MM KK5 ue   %OOE599M+.}/@/@/L/L/P/PQVXZ/[/[+\+\(jquMMM 14[1A1A-.)?;;;  %%%4CFF44  T9%%%#, )?;;; )$$$$$$$$%s2(( F"6D+F"" H+,A4H& H+&H+urlsformatc v Kddlm}ddlm}g}|D]}t |}||sQ|||s3||s|||rt jdddcSt|}|rt jdd|ddcS| ||||d d ddddggd } t d t|g} g} |D]G} t| d {Vs| | d d dd2| | H| sg} n1t}tddlm}m}|r ||nd }||s\|5|s!t jd|jdddS|}|t jddddSt d|jt| dd l}||jr|| |d {V} n"t3j|j| |d {V} | r| | z} d| i}t|dg}t d||| d<tt j|| d<||n t9} t;dt=t?|d}n#t@tBf$r tD}YnwxYw| d d|dgD]3}|dr|dd } |d d p|d!d }|s^tG|}tI|| |\}}||d!<|r| d"xxd#z cc<| d$ | t|t|d%t d&| t|t| t d'| t|5d(|dgD}d|i}|dgkrtKd)}nt j|d*d+}tG|}t|| d,<| d d-tL'd.| tL(|S#tR$r}d/tU|} t+d0| | | d<tL'd.| tL(tK| cYd }~Sd }~wwxYw)1a Extract content from specific web pages using available extraction API backend. Returns clean page content (markdown/text) with NO LLM summarization. The extract backends (Firecrawl, Tavily, Exa, Parallel) already return clean, boilerplate-stripped content, so we return it directly and fast. Pages over ``char_limit`` are head+tail truncated with an explicit footer; the full text is stored under cache/web and the footer tells the model how to read_file the omitted middle. Inline base64 images are replaced with ``[IMAGE: alt]`` placeholders (real image URLs are preserved as links). Args: urls (List[str]): List of URLs to extract content from format (str): Desired output format ("markdown" or "html", optional) char_limit (Optional[int]): Per-page char budget sent to the model (default: web.extract_char_limit or 15000). Larger pages truncate. Security: URLs are checked for embedded secrets before fetching. Returns: str: JSON string with a ``results`` list; each entry has ``url``, ``title``, ``content``, ``error``. ``content`` is the (possibly truncated) clean page text. Raises: Exception: If extraction fails or API key is not set r) _PREFIX_RE)unquoteFz_Blocked: URL contains what appears to be an API key or token. Secrets must not be sent in URLs.rz9Blocked: URL contains a credential-like query parameter (z). Web extract backends are third-party readers; remove the sensitive query parameter or use a local browser session when this access is explicitly required.)rrrN)rrpages_extractedpages_truncatedrrtruncation_metricsprocessing_appliedz!Extracting content from %d URL(s)r(z:Blocked: URL targets a private or internal network addressrtitlerr)get_active_extract_providerrGzy is a search-only backend and cannot extract URL content. Set web.extract_backend to firecrawl, tavily, exa, or parallel.)rzcNo web extract provider configured. Set web.extract_backend to firecrawl, tavily, exa, or parallel.zWeb extract via %s: %d URL(s))rresultszExtracted content from %d pagesrrrrrtruncate_and_storerr raw_contentrrrr)r original_size sent_sizez%s (truncated %d -> %d chars)z%s (%d chars, whole)cg|]h}|dd|dd|dd|ddd|vr d|diniiS)rr(r rrrblocked_by_policy)r:).0rs r/ z$web_extract_tool..s     uuUB''w++55B//w    GZ]^F^F^)1-@+ABBdf      r5z%Content was inaccessible or not foundrrrbase64_image_conversionweb_extract_toolzError extracting content: r), agent.redactrrrr"rerrr#rrIrrr!rlrrHr rGsupports_extract display_namer$inspectiscoroutinefunctionrkasyncio to_threadr:rrrrrrrrrrrrrr*rrJ)!rrrrrnormalized_urls_urlnormalized_urlsensitive_query_keyr safe_urls ssrf_blockedrr rEr rrPrresponsereffective_char_limitresultr cleanr truncatedtrimmed_resultstrimmed_responsercleaned_resultrrs! r/rrsD('''''$$$$$$!#O//2488   d # #   //   00   !8!899  : =    9HH  : O+OOO    ~....$$  "#     Oi% 7_9M9MNNN -/ " & &C*3//////// &##Y%%   %%%%C GG*,,G ' ( ( (         6=F((111$Hx'@'@'B'B'0I0I0K0K':',#+#8!<!<!<&+    7688#:',!<&+     KK/I    NNN**8+;<<  ( 0 06 0 J JJJJJJJ!( 1$i!!!  -"W,Gw'hll9b99:: 5GGG-<)*47 88L8L4M4M01-7-CzzI`IbIb >#&tS5I1J1JG-T-T#U#U :& > > >#=  > ,-445IJJJll9b11 E EFzz'"" **UB''C **]B77T6::iQS;T;TK 2;??E$9%FZ$[$[ !J *F9  E 1222a7222 45<<%(ZZ!$Z>>  ;S#e**cR\oo^^^^ 2CUDDDD   \\)R00    &7    * *b 0 0$%LMMKK*%5aeTTTK 8 DD14^1D1D-.,-445NOOO *O<<<  %%%9Q99  T9%%%#, *O<<< )$$$$$$$$%sR7C;V/3$V/DV/)+MV/M0-V//M00H>V// X89A4X3-X83X8ctdd}|rt |rdSt dt DrdS ddlm}m }|dup |duS#t$r&}t d|Yd}~d Sd}~wwxYw) uCheck whether the configured web backend is available. Used as the ``check_fn`` gate for the ``web_search`` and ``web_extract`` tool registry entries — so a plugin-registered provider that reports ``is_available()`` must light the tools up even when no built-in backend has credentials (issues #28651, #31873). Resolution funnels through :func:`_is_backend_available`, which delegates non-legacy names to the registry. rEr(Tc34K|]}t|VdSr2)rp)rrEs r/ z$check_web_api_key..s+ N Ng  ) ) N N N N N Nr5r)rr Nz3web provider registry availability check failed: %sF) r<r:r]r-rpanyr^rHrr r*rIrJ)r`rr rKs r/check_web_api_keyr0s"##'' 266<<>>DDFFJ+J77t N N9M N N NNNt         ' & ( ( 4 9**,,D8   JCPPPuuuuus;B C %CC __main__u 🌐 Standalone Web Tools Modulez(========================================rYr(rZu✅ Web backend: r=z! Using Exa API (https://exa.ai)rBz+ Using Parallel API (https://parallel.ai)r@z( Using Tavily API (https://tavily.com)rAz Using SearXNG (search only): r[rDz- Using Brave Search free tier (search only)r?z2 Using DuckDuckGo via ddgs package (search only)z Using self-hosted Firecrawl: /z# Using direct Firecrawl cloud APIz! Using Firecrawl tool-gateway: z0 Firecrawl backend selected but not configuredu$❌ No web search backend configuredzWSet EXA_API_KEY, PARALLEL_API_KEY, TAVILY_API_KEY, FIRECRAWL_API_KEY, FIRECRAWL_API_URLru!🛠️ Web tools ready for use!z Extract char limit: zE chars (pages over this are truncated; full text stored in cache/web)u&🐛 Debug mode ENABLED - Session ID: z Debug logs will be saved to: z/web_tools_debug_z.jsonu=🐛 Debug mode disabled (set WEB_TOOLS_DEBUG=true to enable)z Basic usage:z9 from web_tools import web_search_tool, web_extract_toolz import asyncioz # Search (synchronous)z/ results = web_search_tool('Python tutorials')u9 # Extract (asynchronous, no LLM — truncate-and-store)z async def main():z? content = await web_extract_tool(['https://example.com'])z# # bigger budget for one call:zU content = await web_extract_tool(['https://docs.python.org'], char_limit=40000)z asyncio.run(main())z Debug mode:z export WEB_TOOLS_DEBUG=truez3 # Logs saved to: ./logs/web_tools_debug_UUID.json)registryr web_searchaSearch the web for information. Returns up to 5 results by default with titles, URLs, and descriptions. The query is passed through to the configured backend, so operators such as site:domain, filetype:pdf, intitle:word, -term, and "exact phrase" may work when the backend supports them.objectstringzThe search query to look up on the web. You may include backend-supported operators such as site:example.com, filetype:pdf, intitle:word, -term, or "exact phrase".)type descriptionintegerz3Maximum number of results to return. Defaults to 5.r)r7r8minimummaximumdefaultr)r7 propertiesrequired)r$r8r web_extractu+Extract content from web page URLs. Returns clean page content in markdown/text (no LLM summarization — fast). Also works with PDF URLs (arxiv papers, documents) — pass the PDF link directly. Pages within the char budget (default 15000) return whole; larger pages return a head+tail window with a footer telling you the full text's saved file path and the read_file call to page through the omitted middle. Inline images appear as [IMAGE: alt] placeholders; real image URLs are kept as links. If a URL fails or times out, use the browser tool instead.arrayr7z:List of URLs to extract content from (max 5 URLs per call))r7itemsr8maxItemszOptional per-page character budget sent back (default 15000). Pages larger than this are head+tail truncated with the full text stored to disk. Raise it when you need more of a long page inline.r)r7r8r:)rrr9c tt|dd|ddS)Nrr(rr)r)rr:argskws r/rGcs1txx/D/DDHHU\^_L`L`aaar5u🔍i)r$toolsetschemahandlercheck_fn requires_envemojimax_result_size_charsc tt|dtr|dgddngd|dS)Nrrmarkdownr)r)r isinstancer:listrDs r/rGrGmsb/$.txx/?/?$F$FNRaR  B88L))   r5Tu📄) r$rHrIrJrKrLis_asyncrMrN)r%N)r)NN)l__doc__rloggingr+rrtypingrrrrrhttpxrCrplugins.web.firecrawl.providerr r r r r plugins.web.tavily.providerrrrplugins.web.parallel.providerrrplugins.web.exa.providerrr__annotations__rrrrtools.debug_helpersrtools.managed_tool_gatewayrr_peek_nous_access_tokenr_read_nous_access_tokenrtools.tool_backend_helpersrrr tools.url_safetyr!r"r#sys getLogger__name__rIrr0r3r4dictr< frozensetr^rLrQrTrcrirlrgrpr_rRr}rrrrrrrtuplerrrrr0print web_availabletool_gateway_availabler,r-firecrawl_key_availablefirecrawl_url_availablerErstripexitactive session_idlog_dirtools.registryr3rWEB_SEARCH_SCHEMAWEB_EXTRACT_SCHEMAregisterrhr5r/rws ###J  ;;;;;;;;;;;;;; $######544444 $(8C='''*.(3-..."&(3-&&&(, ,,,! Xc]!!!,,,,,,  feeeeeeeee  8 $ $ SS("3"4""""$*!yVVV c&"/c////d -S - - - -.c....      )3)4))))X $    049V#" k+< = = = & & & & &