_j@SrSSKrSSKJrJrJrJrJrJr SSK J r J r J r J r JrJrJr SSKJrJrJrJr SSKJr SSK Jr S \\\S \\4S jrS \\\4S \\\44S jrSSSSSSSSS.S\S\\S\\ S\\\\4S\\S\\S\\S\\S\\S\\S \ 4SjjrS0S\S\S\\S \ 4SjjrSS.S\S\S\\ S \ 4S jjr!S0S\S\S!\\S\\S \\4 S"jjr"S\S\S \4S#jr#S1S\S\S$\S%\\S \ 4 S&jjr$SSSSSSSSSSS'. S\S\\S\\ S\\\\4S\\S\\S\\S\\S\\S\\S$\S%\\S \ 4S(jjr%S\\S \\4S)jr&SSSSSSSS*.S\\S\\ S\\\\4S\\S\\S\\S\\S\\S \'4S+jjr(S2S\\S,\S \\\4S-jjr)S3S\S\\S\\ S,\S$\S%\\S \\4S.jjr*S\S\S \4S/jr+g)4z4 Batch scraping functionality for Firecrawl v2 API. N)OptionalListCallableDictAnyUnion)BatchScrapeRequestBatchScrapeResponseBatchScrapeJob ScrapeOptionsDocument WebhookConfigPaginationConfig) HttpClienthandle_response_errorvalidate_scrape_optionsprepare_scrape_options)normalize_document_input)CrawlErrorsResponse data_listreturnc/nU=(d /H?n[U[5(dM[U5nUR[ S0UD65 MA U$)N) isinstancedictrappendr)r documentsdoc normalizeds O/home/rurouni/.local/lib/python3.13/site-packages/firecrawl/v2/methods/batch.py_parse_batch_scrape_documentsr"sL "IB c4 1#6J   X3 3 4 bodyc tURS5(d[URSS55eURS5URSS5URSS5URS5URS 5URS 5[URS /5=(d /5S .$) NsuccesserrorUnknown error occurredstatus completedrtotal creditsUsed expiresAtnextdatar)r*r+ credits_used expires_atr.r/)get Exceptionr")r$s r!#_parse_batch_scrape_status_responser5s 88I  *BCDD((8$XXk1-'1%/hh{+ -dhhvr.B.HbI r#optionswebhook append_to_idignore_invalid_urlsmax_concurrencyzero_data_retention integrationidempotency_keyclienturlsr7r8r9r:r;r<r=r>c [UUUUUUUUS9n URU 5n URSXS9n U R(d [ U S5 U R 5n U R S5(d[U R SS55e[U R S5U R S 5U R S 5=(d S S 9$) a# Start a batch scrape job for multiple URLs. Args: client: HTTP client instance urls: List of URLs to scrape options: Scraping options Returns: BatchScrapeResponse containing job information Raises: FirecrawlError: If the batch scrape operation fails to start r7r8r9r:r;r<r=z/v2/batch/scrape)headerszstart batch scraper&r'r(idurl invalidURLsN)rDrE invalid_urls) prepare_batch_scrape_request_prepare_headerspostokrjsonr3r4r )r?r@r7r8r9r:r;r<r=r> request_datarCresponser$s r!start_batch_scraperO-s80 !/'/ L%%o6G{{-|{MH ;;h(<= ==?D 88I  *BCDD  88D> HHUOXXm,4 r#job_idpagination_configc lURSU35nUR(d [US5 UR5n[ U5nUSnU(a UR OSnU(aUS(a[ UUSUU5n[USUSUSUS US U(dUSUS 9$S US 9$) a7 Get the status of a batch scrape job. Args: client: HTTP client instance job_id: ID of the batch scrape job pagination_config: Optional configuration for pagination behavior Returns: BatchScrapeJob containing job status and data Raises: FirecrawlError: If the status check fails /v2/batch/scrape/zget batch scrape statusr/Tr.r)r*r+r1r2Nr0)r3rKrrLr5 auto_paginate_fetch_all_batch_pagesr )r?rPrQrNr$payloadrrTs r!get_batch_scrape_statusrWgs(zz-fX67H ;;h(AB ==?D1$7GI8I%33dM*  FO     x +&g^,<($1WV_   8<  r#)request_timeoutnext_urlrXc URXS9nUR(d [US5 UR5n[ U5n[ USUSUSUSUSUSUS S 9$) a Fetch a single page of batch scrape results using the provided next URL. Args: client: HTTP client instance next_url: Opaque next URL from a prior batch scrape status response request_timeout: Timeout (in seconds) for the HTTP request Returns: BatchScrapeJob with the page data and next URL (if any) Raises: Exception: If the request fails or returns an error response )timeoutzget batch scrape status pager)r*r+r1r2r.r/r0)r3rKrrLr5r )r?rYrXrNr$rVs r!get_batch_scrape_status_pager\s~(zz(z H E  %DNN,cN d . +MMO  >yIL %V,H&3y>[+H   X & -  "s9~'D  #6* a K +N %  " % s E** E87E8cURSU35nUR(d [US5 UR5nUR S5S:H$)z Cancel a running batch scrape job. Args: client: HTTP client instance job_id: ID of the batch scrape job to cancel Returns: BatchScrapeStatusResponse with updated status Raises: FirecrawlError: If the cancellation fails rSzcancel batch scraper) cancelled)deleterKrrLr3)r?rPrNr$s r!cancel_batch_scraperw sO$}}09:H ;;h(=> ==?D 88H  ,,r# poll_intervalr[c[R"5n[X5nURS;aU$U(a.[R"5U- U:a[ SUSUS35e[R "U5 Mj)a Wait for a batch scrape job to complete, polling for status updates. Args: client: HTTP client instance job_id: ID of the batch scrape job poll_interval: Seconds between status checks timeout: Maximum seconds to wait (None for no timeout) Returns: BatchScrapeStatusResponse when job completes Raises: FirecrawlError: If the job fails or timeout is reached TimeoutError: If timeout is reached )r*failedruzBatch scrape job z did not complete within z seconds)rgrhrWr) TimeoutErrorsleep)r?rPrxr[ro status_jobs r!wait_for_batch_completionr~&s~,!J ,V<     D D  (:5@!26(:ST[S\\def f =! r#) r7r8r9r:r;r<r=r>rxr[c V[UUUUUUUUUU S9 n U Rn [X X5$)a Start a batch scrape job and wait for it to complete. Args: client: HTTP client instance urls: List of URLs to scrape options: Scraping options poll_interval: Seconds between status checks timeout: Maximum seconds to wait (None for no timeout) Returns: BatchScrapeStatusResponse when job completes Raises: FirecrawlError: If the batch scrape fails to start or complete TimeoutError: If timeout is reached r6)rOrDr~)r?r@r7r8r9r:r;r<r=r>rxr[startrPs r! batch_scraperMsLB  !/'/' EXXF %  r#cFU(d [S5e/nUHnU(a[U[5(d[SU35eURS5(d$URS5(d[SU35eUR UR 55 M U$)z Validate and normalize a list of URLs for batch scraping. Args: urls: List of URLs to validate Returns: Validated list of URLs Raises: ValueError: If URLs are invalid zURLs list cannot be emptyz Invalid URL: zhttp://zhttps://z)URL must start with http:// or https://: ) ValueErrorrstr startswithrstrip)r@validated_urlsrEs r!validate_batch_urlsrs 455N*S#..}SE23 3y))S^^J-G-GHNO Ociik* r#rBcL[U5nSU0n U(a#[U5n U (aU RU 5 Ub,[U[5(aX)S'OUR SS9U S'UbX9S'UbXIS'UbXYS'UbXiS'Ub[ U5R 5U S 'U $) z Prepare a batch scrape request payload. Args: urls: List of URLs to scrape options: Scraping options Returns: Request payload dictionary r@r8T) exclude_none appendToIdignoreInvalidURLsmaxConcurrencyzeroDataRetentionr=)rrupdaterr model_dumpr) r@r7r8r9r:r;r<r=rrM scrape_datas r!rHrHs*).N$*N#;L,W5     , gs # #&- #&-&8&8d&8&KL #%1\"&,?()")8%&&,?()&)+&6&<&<&> ]# r# chunk_sizecn/n[S[U5U5HnURXX1-5 M U$)z Split a large list of URLs into smaller chunks for batch processing. Args: urls: List of URLs to chunk chunk_size: Maximum size of each chunk Returns: List of URL chunks r)rangerlr)r@rchunksis r! chunk_urlsrs8F 1c$i , dQ^,-- Mr#c [X5n/nSnUHAn [UU UUUS9n U R(aURU R5 US- nMC U$)a Process a large batch of URLs by splitting into smaller chunks. Args: client: HTTP client instance urls: List of URLs to scrape options: Scraping options chunk_size: Size of each batch chunk poll_interval: Seconds between status checks timeout: Maximum seconds to wait per chunk Returns: List of all scraped documents Raises: FirecrawlError: If any chunk fails r)r7rxr[rb)rrr/extend) r?r@r7rrxr[ url_chunks all_documentscompleted_chunkschunkresults r!process_large_batchrsj2D-JM  '   ;;   -A r#c*URSUS35nUR(d [US5 UR5nURSU5nURS/5URSURS/55S.n[ S 0UD6$) z Get errors for a batch scrape job. Args: client: HTTP client instance job_id: ID of the batch scrape job Returns: CrawlErrorsResponse with errors and robots-blocked URLs rSz/errorszget batch scrape errorsr/errors robotsBlockedrobots_blocked)rrr)r3rKrrLr)r?rPrNr$rVr s r!get_batch_scrape_errorsrszz-fXW=>H ;;h(AB ==?Dhhvt$G++h+!++ow{{CSUW7XYJ  , ,,r#)N)r N)d)Nrr N),__doc__rgtypingrrrrrrtypesr r r r rrrutilsrrrrutils.normalizerrr"rr5boolintrOrWfloatr\rUrwr~rrrrHrrrrr#r!rs ==gf6'Xd3i-@T(^ d38n c3h &(,37"&*.%)*.!%%)7 7 s)7m $ 7 eC./ 0 7 3- 7"$7c]7"$7#7c]77z591 1 1 011 1p(, $ $$e_ $  $V59 D DDH~D 01 D  (^ DN- - - -@! $" $" $"$"c] $"  $"V(,37"&*.%)*.!%%)!3 3 s)3m $ 3 eC./ 0 3 3- 3"$3c]3"$3#3c]33c]33ld3iDIB(,37"&*.%)*.!%/ s)/m $/eC./ 0 / 3- / "$ /c]/"$/#/ /dT#YC$tCy/((,! - - s)-m $- -  - c] - (^-`-J--@S-r#