_j! SSKJrJrJrJr SSKJrJrJrJ r J r J r SSK J r SSKJr SSKJr SSKJr SSKJr SS KrS \\\S \\4S jrS \\\4S \\\44SjrS S.S\\S\\S \\\44SjjrS\ S\\S \ 4SjrS S\ S\S\\ S \ 4SjjrS S.S\ S\S\\S \ 4SjjrS S\ S\S\\S\\ S \\4 SjjrS\ S\S \ 4Sjr!S\ S\S \\\44Sjr"g )!)OptionalListDictAny) ScrapeOptions WebhookConfigDocumentBatchScrapeResponseBatchScrapeJobPaginationConfig)AsyncHttpClient)prepare_scrape_options)handle_response_error)normalize_document_input)validate_batch_urlsN data_listreturnc/nU=(d /H?n[U[5(dM[U5nUR[ S0UD65 MA U$)N) isinstancedictrappendr )r documentsdoc normalizeds S/home/rurouni/.local/lib/python3.13/site-packages/firecrawl/v2/methods/aio/batch.py_parse_batch_scrape_documentsr sL "IB c4 1#6J   X3 3 4 bodyc tURS5(d[URSS55eURS5URSS5URSS5URS5URS 5URS 5[URS /5=(d /5S .$) NsuccesserrorUnknown error occurredstatus completedrtotal creditsUsed expiresAtnextdatar%r&r' credits_used expires_atr*r+)get Exceptionr)r s r#_parse_batch_scrape_status_responser1s 88I  *BCDD((8$XXk1-'1%/hh{+ -dhhvr.B.HbI r)optionsurlsr2c U(d [S5e[UVs/sH*n[U[5(aUR 5OUPM, sn5nSU0nU(a#[ U5nU(aUR U5 URS5=nb)[U[5(aUOURSS9US'URS5=nbXS'URS5=nbXS 'URS 5=nbXS 'URS 5=nbXS 'URS5=nb$[U5R 5n U (aXS'U$s snf)NzURLs list cannot be emptyr3webhookT) exclude_none append_to_id appendToIdignore_invalid_urlsignoreInvalidURLsmax_concurrencymaxConcurrencyzero_data_retentionzeroDataRetention integration) ValueErrorrrstrstriprupdater/ model_dump) r3r2kwargsuvalidated_urlspayloadoptswvtrimmed_integrations r_preparerM!sS 455(Z^)_Z^UVz!S7I7I!'')q*PZ^)_`N%~6G%g.  NN4 ZZ ""/",Q"4"4Q!,,TX,:Y  ZZ ''4 !  ZZ- ..;'(#$ ZZ) **7$% ! ZZ- ..;'(#$ ZZ &&3!!flln %8M " N)*`s1Eclientc# [U40UD6nURSU5IShvN nURS:a [US5 UR 5nUR S5(d[ UR SS55e[UR S5UR S5UR S 5S 9$N7f) Nz/v2/batch/scrapezstart batch scraper"r#r$idurl invalidURLs)rQrR invalid_urls)rMpost status_coderjsonr/r0r )rNr3rErHresponser s rstart_batch_scraperY<st&v&G[[!3W==Hs"h(<= ==?D 88I  *BCDD $((4.dhhuoTXT\T\]jTk ll >s"CB>BCjob_idpagination_configc # URSU35IShvN nURS:a [US5 UR5n[ U5nUSnU(a UR OSnU(a#US(a[ UUSUU5IShvN n[USUS US US US U(dUSUS 9$SUS 9$NN37f)a8 Get the status of a batch scrape job. Args: client: Async HTTP client instance job_id: ID of the batch scrape job pagination_config: Optional configuration for pagination behavior Returns: BatchScrapeJob containing job status and data Raises: Exception: If the status check fails /v2/batch/scrape/NrPzget batch scrape statusr+Tr*r%r&r'r-r.r,)r/rVrrWr1 auto_paginate_fetch_all_batch_pages_asyncr )rNrZr[rXr rHdocsr^s rget_batch_scrape_statusraGs&ZZ"3F8 <==Hs"h(AB ==?D1$7G 6?D8I%33dM1  FO      x +&g^,<($1WV_   8<  #> s"C CA:C C 2C  C )request_timeoutnext_urlrbc # URXS9IShvN nURS:a [US5 UR5n[ U5n[ USUSUSUSUS US US S 9$N_7f) a Fetch a single page of batch scrape results using the provided next URL. Args: client: Async HTTP client instance next_url: Opaque next URL from a prior batch scrape status response request_timeout: Timeout (in seconds) for the HTTP request Returns: BatchScrapeJob with the page data and next URL (if any) Raises: Exception: If the request fails or returns an error response )timeoutNrPzget batch scrape status pager%r&r'r-r.r*r+r,)r/rVrrWr1r )rNrcrbrXr rHs rget_batch_scrape_status_pagerfvs(ZZZBBHs"h(FG ==?D1$7G x +&g^,<( V_ V_  CsA7A5A A7initial_documentsc# UR5nUnSnU(a UROSnU(a UROSnU(a UROSn [R "5n U(GaUbXg:aU$U b[R "5U - U :aU$UR U5IShvN n U RS:a6SSKn U RS5n U RSU R35 U$U R5n[U5nUSH(nUb[U5U:a OURU5 M* Ub[U5U:aU$USnUS- nU(aGMU$N![a U$f=f7f) a; Fetch all pages of batch scrape results asynchronously. Args: client: Async HTTP client instance next_url: URL for the next page initial_documents: Documents from the first page pagination_config: Optional configuration for pagination limits Returns: List of all documents from all pages rNrP firecrawlzFailed to fetch next page: r+r*)copy max_pages max_results max_wait_timetime monotonicr/rVlogging getLoggerwarningrWr1r0lenr)rNrcrgr[r current_url page_countrlrmrn start_timerXrqlogger page_data page_payloaddocuments rr_r_s$"&&(IKJ0A!++dI3D#//$K7H%33dM!J   ! (? H E  %DNN,yIL %V,H'c)n .K   X & -  ##i.K*G  #6* a K +N =1  " % s>B2F 4E75AF  E9AF 5F 9 FF FF c# URSU35IShvN nURS:a [US5 UR5nUR S5S:H$ND7f)Nr]rPzcancel batch scraper% cancelled)deleterVrrWr/rNrZrXr s rcancel_batch_scrapers\]]%6vh#?@@Hs"h(=> ==?D 88H  ,, AsA!AAA!c# URSUS35IShvN nURS:a [US5 UR5nURS5(d[ URSS55eU$Nc7f)Nr]z/errorsrPzget batch scrape errorsr"r#r$)r/rVrrWr0rs rget_batch_scrape_errorsrsuZZ"3F87 CDDHs"h(AB ==?D 88I  *BCDD K EsBA?A$B)N)#typingrrrrtypesrr r r r r utils.http_client_asyncrutils.validationrutils.error_handlerrutils.normalizer methods.batchrrorrAr1rMrYrafloatrfr_boolrrrrrrs,,rr66870 Xd3i-@T(^ d38n c3h EI49(=*AW[\_ad\dWe6m_mDImTgm59, , , 01, ,f(, ! !!e_ !  !P59 D DDH~D 01 D  (^ DN-o-s-t-/34PSUXPX>r