_j2TSrSSKrSSKJrJrJrJr SSKJrJ r J r J r J r J r JrJrJrJrJrJr SSKJrJrJrJr SSKJr S\S S4S jrS\S \4S jrS \\\S \\ 4S jrS\\\4S \\\44Sjr S\S\S \ 4Sjr!S%SS.S\S\S\\S\\"S \ 4 Sjjjr#SS.S\S\S\\"S \ 4Sjjr$S%SS.S\S\S\\ S\\S\\"S \\ 4 Sjjjr%S\S\S \&4Sjr'S&SS.S\S\S\(S\\(S\\"S \ 4 Sjjjr)S&SS.S\S\S\(S\\(S\\"S \ 4 Sjjjr*S\S\ S \4S jr+S!\S"\S \4S#jr,S\S \4S$jr-g)'z. Crawling functionality for Firecrawl v2 API. N)OptionalDictAnyList) CrawlRequestCrawlJob CrawlResponseDocumentCrawlParamsRequestCrawlParamsResponseCrawlParamsData WebhookConfigCrawlErrorsResponseActiveCrawlsResponse ActiveCrawlPaginationConfig) HttpClienthandle_response_errorvalidate_scrape_optionsprepare_scrape_options)normalize_document_inputrequestreturncUR(aURR5(d [S5eURbURS::a [S5eURb[ UR5 gg)z Validate crawl request parameters. Args: request: CrawlRequest to validate Raises: ValueError: If request is invalid URL cannot be emptyNrzLimit must be positive)urlstrip ValueErrorlimitscrape_optionsr)rs O/home/rurouni/.local/lib/python3.13/site-packages/firecrawl/v2/methods/crawl.py_validate_crawl_requestr#sk ;;gkk//11.//}} W]]a%7122) 6 67*c"[U5 SUR0nUR(aURUS'URb [ UR5nU(aX!S'UR SSS9nUR SS5 UR SS5 UR SS5 URbK[UR[5(aURUS'OURR SS 9US'S S S S SSSSSSSSSSSS.nUR5HupVXS;dM UR U5X'M! URU5 SU;a.[US[5(aUSR5US'U$)z Prepare crawl request for API submission. Args: request: CrawlRequest to prepare Returns: Dictionary ready for API submission rpromptN scrapeOptionsT) exclude_none exclude_unsetr!webhook)r( includePaths excludePathsmaxDiscoveryDepthsitemapignoreQueryParametersdeduplicateSimilarURLscrawlEntireDomainallowExternalLinksallowSubdomainsignoreRobotsTxtrobotsUserAgentdelaymaxConcurrencyregexOnFullURLzeroDataRetention) include_paths exclude_pathsmax_discovery_depthr.ignore_query_parametersdeduplicate_similar_urlscrawl_entire_domainallow_external_linksallow_subdomainsignore_robots_txtrobots_user_agentr6max_concurrencyregex_on_full_urlzero_data_retention integration) r#rr&r!r model_dumppopr* isinstancestritemsupdater)rdata scrape_data request_datafield_mappings snake_case camel_cases r"_prepare_crawl_requestrT&sG$ 7;; D~~ X),W-C-CD $/ !%%4t%LLUD!Xt$%t," goos + +%ooDO&oo88d8KDO('2#:$<2 4-..+-2N&#1"6"6"8  %+// ;D #9  KK D,?!E!E"=1779] Kr$ data_listc /nU=(d /H=n[U[5(dMUR[S0[ U5D65 M? U$)N)rJdictappendr r)rU documentsdoc_datas r"_parse_crawl_documentsr\qsH "IOO h % %   XK(@(JK L$ r$ response_datac dURS5(d[URSS55eURS5URSS5URSS5URSS5URS 5URS 5[URS /55S .$) NsuccesserrorUnknown error occurredstatus completedrtotal creditsUsed expiresAtnextrNrbrcrd credits_used expires_atrgrN)get Exceptionr\)r]s r"_parse_crawl_status_responsermys   Y ' ' ))'3KLMM ##H-"&&{A6""7A.%))-;#'' 4!!&)&}'8'8'DE r$clientcT[U5nURSU5nUR(d [US5 UR 5nUR S5(a.UR S5UR S5S.n[ S 0UD6$[UR SS55e) a Start a crawl job for a website. Args: client: HTTP client instance request: CrawlRequest containing URL and options Returns: CrawlResponse with job information Raises: ValueError: If request is invalid Exception: If the crawl operation fails to start z /v2/crawlz start crawlr_idr)rprr`rarW)rTpostokrjsonrkr rl)rnrrPresponser]job_datas r" start_crawlrvs*'2L{{; 5H ;;h 6MMOM####D) $$U+  (x(( ))'3KLMMr$request_timeoutjob_idpagination_configrxc URSU3US9nUR(d [US5 UR5n[ U5nUSnU(a UR OSnU(aGUS(a=U(a&UR b[U5UR :d[UUSUUUS9n[US US US US US U(dUSUS9$SUS9$)a Get the status of a crawl job. Args: client: HTTP client instance job_id: ID of the crawl job pagination_config: Optional configuration for pagination behavior request_timeout: Timeout (in seconds) for each individual HTTP request. When auto-pagination is enabled (default) and there are multiple pages of results, this timeout applies to each page request separately, not to the entire operation Returns: CrawlJob with current status and data Raises: Exception: If the status check fails /v2/crawl/timeoutzget crawl statusrNTrgNrwrbrcrdrirjrh) rkrrrrsrm auto_paginate max_resultslen_fetch_all_pagesr ) rnryrzrxrtr]payloadrZrs r"get_crawl_statusrs2zzJvh/zIH ;;h(:;MMOM*=9GI8I%33dM  ) ) 5  N/;; ;$  FO  +   x +&g^,<($1WV_   8<  r$next_urlc URXS9nUR(d [US5 UR5n[ U5n[ USUSUSUSUSUSUS S 9$) au Fetch a single page of crawl results using the provided next URL. Args: client: HTTP client instance next_url: Opaque next URL from a prior crawl status response request_timeout: Timeout (in seconds) for the HTTP request Returns: CrawlJob with the page data and next URL (if any) Raises: Exception: If the request fails or returns an error response r}zget crawl status pagerbrcrdrirjrgrNrh)rkrrrrsrmr )rnrrxrtr]rs r"get_crawl_status_pagers~(zz(z J G  %DNN,cN d 0 -MMO  7 BL %V,H&3y>[+H   X & -  "s9~'D  #6* a M +P %  " % s E(( E65E6cURSU35nUR(d [US5 UR5nUR S5S:H$)z Cancel a running crawl job. Args: client: HTTP client instance job_id: ID of the crawl job to cancel Returns: bool: True if the crawl was cancelled, False otherwise Raises: Exception: If the cancellation fails r|z cancel crawlrb cancelled)deleterrrrsrk)rnryrtr]s r" cancel_crawlr^sK}}z&23H ;;h7MMOM   X &+ 55r$ poll_intervalr~c[R"5n[UUUS9nURS;aU$Ub.[R"5U- U:a[ SUSUS35e[R "U5 Mf)a Wait for a crawl job to complete, polling for status updates. Args: client: HTTP client instance job_id: ID of the crawl job poll_interval: Seconds between status checks timeout: Maximum seconds to wait (None for no timeout) request_timeout: Optional timeout (in seconds) for each status request Returns: CrawlJob when job completes Raises: Exception: If the job fails TimeoutError: If timeout is reached rw)rcfailedrz Crawl job z did not complete within z seconds)rrrrb TimeoutErrorsleep)rnryrr~rxr crawl_jobs r"wait_for_crawl_completionrus2!J $  +    C C   DNN$4z$AW#LF83LWIU]^_ _ =!! r$cX[X5nURnUbUOUn[UUUUUS9$)a Start a crawl job and wait for it to complete. Args: client: HTTP client instance request: CrawlRequest containing URL and options poll_interval: Seconds between status checks timeout: Maximum seconds to wait for the entire crawl job to complete (None for no timeout) request_timeout: Timeout (in seconds) for each individual HTTP request, including pagination requests when fetching results. If there are multiple pages, each page request gets this timeout Returns: CrawlJob when job completes Raises: ValueError: If request is invalid Exception: If the crawl fails to start or complete TimeoutError: If timeout is reached rw)rvrpr)rnrrr~rxrryeffective_request_timeouts r"crawlrsF8F,I \\F4C3NT[ %1  r$cUR(aURR5(d [S5eUR(aURR5(d [S5eURURS.nUR SU5nUR (d [ US5 UR5nURS5(GahURS05n0nSS S S S S SSSSSSSSS.nSU;a-USn[U[5(a[S'0UD6US'OXS'UR5HupX;dM U S:XaXYbXYn 0n SSSSSSS.n U R5HupX;dM XX'M S U ;a/U S n[U[5(aS!S"KJn U"US#9U S 'OUU S 'U R5HunnUU ;dMUS :wdMUU U'M XU 'MXYXj'M UR5HunnUU;dMUUU'M S$U;aUS$US$'[!S'0UD6$[#URS%S&55e)(a8 Get crawl parameters from LLM based on URL and prompt. Args: client: HTTP client instance request: CrawlParamsRequest containing URL and prompt Returns: CrawlParamsData containing suggested crawl options Raises: ValueError: If request is invalid Exception: If the operation fails rzPrompt cannot be empty)rr&z/v2/crawl/params-previewzcrawl params previewr_rNr:r;r<r.r=r>r?r@rArBrCrDr!rF)r+r,r-r.r/r0r1r2r3r4r5r7r'r9r*r' include_tags exclude_tagsonly_main_contentwait_forskip_tls_verificationremove_base64_images) includeTags excludeTagsonlyMainContentwaitForskipTlsVerificationremoveBase64Imagesformatsr) ScrapeFormats)rrr`rarW)rrrr&rqrrrrsrkrJrXrrLlisttypesrrrl)rnrrPrtr] params_dataconverted_paramsrQ webhook_datarSrRscrape_opts_dataconverted_scrape_optsscrape_field_mappings scrape_camel scrape_snake formats_datarkeyvalues r"crawl_params_previewrs ;;gkk//11.// >>!5!5!7!7122{{..L {{5|DH ;;h(>?MMOM###''3 ++!6 %>&@!6"8122/-!6 $  #&y1L,--.;.Kl.K +.:+&4&:&:&< "J(0[5L5X'2'>$,.)'5'5+>#-/F.D -)7L6Q6Q6S2 ';BRB`1?7T !$44'7 'B %lD99=?LUa?b1) U&;;y@P9>1#6'?4IZ03>3J$0I'=N&++-JC.((- %.  %*7 *B Y '2!122 ))'3KLMMr$ http_clientcrawl_idclURSUS35nUR(d [US5 UR5nURSU5nURS/5URSURS/55S.n[ S 0UD6$![ an[ S U35eS nAff=f) z Get errors from a crawl job. Args: http_client: HTTP client for making requests crawl_id: The ID of the crawl job Returns: CrawlErrorsResponse containing errors and robots blocked URLs Raises: Exception: If the request fails r|z/errorszcheck crawl errorsrNerrors robotsBlockedrobots_blocked)rrz'Failed to parse crawl errors response: NrW)rkrrrrsrrl)rrrtbodyr normalizedes r"get_crawl_errorsrGsH:W=>H ;;h(<= G}}((64(kk(B/%kk/7;;GWY[;\] #0Z00 GA!EFFGsA!B B3 B..B3c ^URS5nUR(d [US5 UR5nURS5(d[ URSS55eURS/5n/nUH|n[ U[ 5(dMURURS5URSURS 55URS 5URS 5S .5 M~ [S UVs/sHn[S0UD6PM snS9$s snf)z Get a list of currently active crawl jobs. Args: client: HTTP client instance Returns: ActiveCrawlsResponse containing a list of active crawl jobs Raises: Exception: If the request fails z/v2/crawl/activezget active crawlsr_r`racrawlsrpteamIdteam_idroptions)rprrrT)r_rrW) rkrrrrsrlrJrXrYrr)rnrtr crawls_innormalized_crawlscncs r"get_active_crawlsrgszz,-H ;;h(;< ==?D 88I  *BCDD2&I  a    $ $eeDk55155+;<uuU|55+ &  Rc5dRcBk6GB6GRc5d ee5dsD* )N)rN).__doc__rtypingrrrrrrr r r r r rrrrrrutilsrrrrutils.normalizerr#rXrTr\rKrmrvfloatrrrboolrintrrrrrrWr$r"rs ,, gf68\8d8*HLHTHVhtCy&9d8n S#X 4S>  N N\ Nm NL59> (, > > > 01> e_ >  >J(, $ $$e_ $  $V59 H (, H HHH~H 01 H e_ H (^HV66S6T64! +" (, +" +" +"+"c] +" e_ +"+"b! ) (, ) ) ))c] ) e_ ))XuNuN6HuN_uNpG*GG@SG@ fj f-A fr$