_j~ %SrSSKrSSKrSSKrSSKJrJrJrJrJ r J r J r J r J r SSKrSSKJr SSKrSSKrSSKrSSKrSSKrSSKr\R,"S5r\R0\S'Sr\"5r\ "S5r"S S \R:5r"S S \R:5r"S S\R:5r "SS\R:5r!"SS\R:\ \5r""SS\R:5r#"SS\R:5r$"SS\R:5r%"SS\R:5r&"SS\R:5r'"SS\R:5r("SS \R:5r)"S!S"\R:5r*"S#S$\R:5r+"S%S&\R:5r,"S'S(\R:5r-"S)S*\R:5r."S+S,\R:5r/"S-S.\R:5r0"S/S0\R:5r1"S1S2\&5r2"S3S4\"\\ \5r3"S5S6\R:5r4"S7S8\R:5r5"S9S:\R:5r6"S;S<\R:5r7"S=S>\R:5r8"S?S@\R:5r9"SASB\R:5r:"SCSD\R:5r;"SESF\R:5r<"SGSH\R:5r="SISJ\R:\ \5r>"SKSL\R:5r?"SMSN\R:5r@"SOSP\R:5rA"SQSR\R:5rB"SSST\R:5rC"SUSV\R:5rD"SWSX\R:5rE"SYSZ\R:5rF"S[S\\R:5rG"S]S^\R:5rH"S_S`\R:5rI"SaSb\R:5rJ"ScSd\R:5rK"SeSf\R:5rL"SgSh\R:5rM"SiSj\R:5rN"SkSl\R:5rO"SmSN\R:5r@"SnSH\R:5r="SoSp5rP"SqSr5rQ"SsSt\P5rR"SuSv\Q5rSg)wa" Firecrawl v1 API Client - Legacy Implementation This module provides the legacy v1 implementation of the Firecrawl SDK. It contains the complete `V1FirecrawlApp` class with all v1 API methods and types for backward compatibility. This is used by the unified client to provide version-specific access patterns like app.v1.scrape_url(). Classes: - V1FirecrawlApp: Legacy v1 client for interacting with the Firecrawl API. - AsyncV1FirecrawlApp: Async version of the v1 client. - CrawlWatcher: WebSocket-based crawl monitoring for v1. N) AnyDictOptionalListUnionCallableLiteralTypeVarGeneric)datetime firecrawlloggercSSKJn [RR [ 5nU"[RR US55R5n[R"SU[R5nU(aURS5R5$g![a [S5 gf=f)Nr)Pathz __init__.pyz"^__version__ = ['\"]([^'\"]*)['\"]z&Failed to get version from __init__.py)pathlibrospathdirname__file__join read_textresearchMgroupstrip Exceptionprint)r package_path version_file version_matchs H/home/rurouni/.local/lib/python3.13/site-packages/firecrawl/v1/client.py get_versionr$s WW__X.l"'',,|]CDNNPlii E|UWUYUYZm $$Q'--/ /   45 sB(B,,CCTcB\rSrSr%SrSr\S\S'Sr\ \ \S'Sr g)V1AgentOptionsPzConfiguration for the agent.FIRE-1modelNprompt) __name__ __module__ __qualname____firstlineno____doc__r*r __annotations__r+rstr__static_attributes__r,r#r'r'Ps"&'E78 ' FHSM r5r'c.\rSrSr%SrSr\S\S'Srg)V1AgentOptionsExtractU2Configuration for the agent in extract operations.r)r*r,N r-r.r/r0r1r*r r2r4r,r5r#r7r7U<'E78 'r5r7c:\rSrSr%Sr\\\S'\\\S'Srg)V1ActionsResultYz,Result of actions performed during scraping. screenshotspdfsr,N) r-r.r/r0r1rr3r2r4r,r5r#r=r=Ys6c s)Or5r=c\rSrSr%SrSr\\\S'\\S'\\S'Sr \\ \\ 4\S'\ R"SSS 9r\\ \S 'S rg) V1ChangeTrackingData^z& Data for the change tracking format. NpreviousScrapeAt changeStatus visibilitydiffjsonalias json_fieldr,)r-r.r/r0r1rDrr3r2rGrrpydanticFieldrKr4r,r5r#rBrB^sS'+hsm*O%)D(4S> ") (t6 BJ Br5rBcB\rSrSr%SrSr\\\S'Sr \\\S'Sr \\\S'Sr \\\S'Sr \\ \\S'Sr\\\S '\R""SS S 9r\\\S 'Sr\\\S 'Sr\\\S'Sr\\\S'Sr\\\S'Sr\\\S'Sr\\\S'Srg)V1FirecrawlDocumenthz-Document retrieved or processed by Firecrawl.NurlmarkdownhtmlrawHtmllinksextractrHrIrK screenshotmetadataactionstitle descriptionchangeTrackingr,)r-r.r/r0r1rQrr3r2rRrSrTrUrrVr%rLrMrKrWrXrrYr=rZr[r\rBr4r,r5r#rOrOhs7C#"Hhsm"D(3-!GXc]!!%E8DI %GXa[&nnT@J @ $J $"Hhsm")-GXo &-E8C=!%K#%59NH129r5rOcH\rSrSr%SrSr\\\S'Sr \\ \\S'Sr g)V1LocationConfigxz$Location configuration for scraping.Ncountry languagesr,) r-r.r/r0r1r`rr3r2rarr4r,r5r#r^r^xs&.!GXc]!%)IxS ")r5r^c\rSrSr%Sr\\S'Sr\\ \\4\S'Sr \\ \\4\S'Sr \\ \ S\S'S rg) V1WebhookConfig}zConfiguration for webhooks.rQNheadersrX) completedfailedpagestartedeventsr,)r-r.r/r0r1r3r2rerrrXrjrr r4r,r5r#rcrc}sR% H(,GXd38n %,)-HhtCH~&-PTFHT'"JKL MTr5rcc\rSrSr%SrSr\\\S\ S'\ R"SSS9r \\ \ S'Sr\\\ S 'Sr\\\ S 'S rg) V1ChangeTrackingOptionsz"Configuration for change tracking.N)zgit-diffrHmodesschemarI schema_fieldr+tagr,)r-r.r/r0r1rnrrr r2rLrMrprr+r3rqr4r,r5r#rlrlsT,9=E8D!345 6="*..X"FL(3-F FHSM C#r5rlc\rSrSr%SrSr\\\S\ S'Sr \\ \ \ 4\ S'Sr \\\ \ S'Sr\\\ \ S'Sr\\\ S 'Sr\\\ S 'S r\\\ S 'Sr\\\ S 'Sr\\\ S'Sr\\\ S'Sr\\\ S'Sr\\\ S'Sr\\S\ S'Sr\\\ S'Sr\\\ S'Sr\\\ S'Sr\\\ S'Sr g)V1ScrapeOptions#Parameters for scraping operations.N rRrSrTcontentrUrWscreenshot@fullPagerVrHr\formatsre includeTags excludeTagsonlyMainContentwaitFor0utimeoutlocationmobileskipTlsVerificationremoveBase64ImagesblockAdsbasicstealthenhancedautoproxychangeTrackingOptionsmaxAge storeInCacheparsePDFr,)!r-r.r/r0r1ryrrr r2rerr3rzr{r|boolr}intrrr^rrrrrrrlrrrr4r,r5r#rsrss3-eiGXd7$_`abi(,GXd38n %,'+K$s)$+'+K$s)$+&*OXd^*!GXc]!"GXc]"+/Hh'(/!FHTN!*.$.)--#Hhtn#GKE8GBC DK?C8$;<C FHSM #'L(4.'#Hhtn#r5rscR\rSrSr%Sr\S\S'Sr\\ \S'Sr \\ \S'Sr g) V1WaitActionz'Wait action to perform during scraping.waittypeN millisecondsselectorr,) r-r.r/r0r1r r2rrrrr3r4r,r5r#rrs+1 &/"&L(3-&"Hhsm"r5rcR\rSrSr%Sr\S\S'Sr\\ \S'Sr \\ \S'Sr g) V1ScreenshotActionz-Screenshot action to perform during scraping.rWrNfullPagequalityr,) r-r.r/r0r1r r2rrrrrr4r,r5r#rrs,7 , #Hhtn#!GXc]!r5rc4\rSrSr%Sr\S\S'\\S'Srg) V1ClickActionz(Click action to perform during scraping.clickrrr,N r-r.r/r0r1r r2r3r4r,r5r#rrs2 ' Mr5rc4\rSrSr%Sr\S\S'\\S'Srg) V1WriteActionz(Write action to perform during scraping.writertextr,Nrr,r5r#rrs2 '  Ir5rc4\rSrSr%Sr\S\S'\\S'Srg) V1PressActionz(Press action to perform during scraping.pressrkeyr,Nrr,r5r#rrs2 '  Hr5rcN\rSrSr%Sr\S\S'\S\S'Sr\\ \S'S r g) V1ScrollActionz)Scroll action to perform during scraping.scrollr)updown directionNrr,) r-r.r/r0r1r r2rrr3r4r,r5r#rrs)3 ( |$$"Hhsm"r5rc*\rSrSr%Sr\S\S'Srg)V1ScrapeActionz)Scrape action to perform during scraping.scraperr,N)r-r.r/r0r1r r2r4r,r5r#rrs3 ( r5rc4\rSrSr%Sr\S\S'\\S'Srg)V1ExecuteJavascriptActionz5Execute javascript action to perform during scraping.executeJavascriptrscriptr,Nrr,r5r#rrs? % && Kr5rcl\rSrSr%Sr\S\S'Sr\\S\S'Sr \\ \S'Sr \\ \S 'S r g) V1PDFActionz&PDF action to perform during scraping.pdfrN) A0A1A2A3A4A5A6LetterLegalTabloidLedgerformat landscapescaler,)r-r.r/r0r1r r2rrrrrfloatr4r,r5r#rrs@0 %.rvFHWmn ov $Ix~$!E8E?!r5rc.\rSrSr%SrSr\S\S'Srg)V1ExtractAgentr9r)r*r,Nr:r,r5r#rrr;r5rc\rSrSr%SrSr\\\S'\ R"SSS9r \\ \S'Sr \\\S'Sr\\\S 'S rg) V1JsonConfigzConfiguration for extraction.Nr+rorIrp systemPromptagentr,)r-r.r/r0r1r+rr3r2rLrMrprrrrr4r,r5r#rrsI' FHSM "*..X"FL(3-F"&L(3-&&*E8N #*r5rc \rSrSr%SrSr\\\S'Sr \\\S'Sr \\ \ \ \\\\\\\\4 \S'Sr\\\S'Sr\\\S'S rg) V1ScrapeParamsruNrV jsonOptionsrYrwebhookr,)r-r.r/r0r1rVrrr2rrYrrrrrrrrrrrrr'rrcr4r,r5r#rrs-&*GXl #**.K,'.}AGXd5/A=R_anp~AOQjlw"wxyzA&*E8N #*)-GXo &-r5rcP\rSrSr%SrSr\\S'Sr\ \ \S'Sr \ \ \S'Sr g) V1ScrapeResponsez"Response from scraping operations.TsuccessNwarningerrorr,) r-r.r/r0r1rrr2rrr3rr4r,r5r#rrs+,GT!GXc]!E8C=r5rc~\rSrSr%SrSr\\\S'Sr \\\S'Sr \ \S'Sr \\\S'Sr \\\\S 'S rg) V1BatchScrapeResponsez&Response from batch scrape operations.NidrQTrr invalidURLsr,)r-r.r/r0r1rrr3r2rQrrrrrr4r,r5r#rrsL0B C#GTE8C='+K$s)$+r5rc\rSrSr%SrSr\\S'\S\S'\ \S'\ \S'\ \S '\ \S 'S r \ \ \S '\\\S 'Srg )V1BatchScrapeStatusResponsez)Response from batch scrape status checks.Trscrapingrfrg cancelledstatusrftotal creditsUsed expiresAtNnextdatar,r-r.r/r0r1rrr2r rr rrr3rrOr4r,r5r#rrsK3GT B CCN JD(3- " ##r5rc\rSrSr%SrSr\\\\ S'Sr \\\\ S'Sr \\ \ S'Sr \\ \ S'Sr\\ \ S'Sr\\\ S 'Sr\\\ S 'Sr\\\ S 'Sr\\\ S 'Sr\\\ S 'Sr\\\\4\ S'Sr\\\ S'Sr\\\ S'Sr\\\ S'Sr\\ \ S'Sr\\ \ S'Sr\\\ S'Srg) V1CrawlParamsz#Parameters for crawling operations.N includePaths excludePathsmaxDepthmaxDiscoveryDepthlimitallowBackwardLinkscrawlEntireDomainallowExternalLinks ignoreSitemap scrapeOptionsrdeduplicateSimilarURLsignoreQueryParametersregexOnFullURLdelaymaxConcurrencyallowSubdomainsr,) r-r.r/r0r1rrrr3r2rrrrrrrrrrrrsrrrcrrrrrrr4r,r5r#rrs-(,L(49%,(,L(49%,"Hhsm"'+x}+E8C=)--(,x~,)--$(M8D>(/3M8O,359GXeC01 29-1HTN1,08D>0%)NHTN)E8C=$(NHSM(&*OXd^*r5rcd\rSrSr%SrSr\\\S'Sr \\\S'Sr \ \S'Sr \\\S'S r g) V1CrawlResponseiz"Response from crawling operations.NrrQTrrr,)r-r.r/r0r1rrr3r2rQrrrr4r,r5r#rrs9,B C#GTE8C=r5rc\rSrSr%SrSr\\S'\S\S'\ \S'\ \S'\ \S '\ \S 'S r \ \ \S '\\\S 'Srg )V1CrawlStatusResponseiz"Response from crawl status checks.TrrrrfrrrNrrr,rr,r5r#r r sK,GT B CCN JD(3- " ##r5r c`\rSrSr%Sr\\S'Sr\\ \S'\\S'Sr \\\S'\\S'S r g) V1CrawlErrori"zA crawl error.rN timestamprQcoderr,) r-r.r/r0r1r3r2r rr rr4r,r5r#r r "s1 G$(Ix!( HD(3- Jr5r c:\rSrSr%Sr\\\S'\\\S'Sr g)V1CrawlErrorsResponsei*z2Response from crawl/batch scrape error monitoring.errors robotsBlockedr,N) r-r.r/r0r1rr r2r3r4r,r5r#rr*s<  9r5rc\rSrSr%SrSr\\\S'Sr \\ \S'Sr \\ \S'Sr \\ \S'Sr \\\S'S r\\\S 'Sr\\ \S 'Sr\\\S 'S rg) V1MapParamsi/z"Parameters for mapping operations.NrrincludeSubdomains sitemapOnlyrr~ruseIndexrr,)r-r.r/r0r1rrr3r2rrrrrrrrrr^r4r,r5r#rr/sx, FHSM $(M8D>((,x~,"&K$&E8C="GXc]"#Hhtn#+/Hh'(/r5rcV\rSrSr%SrSr\\S'Sr\ \ \ \S'Sr \ \ \S'Sr g) V1MapResponsei:z!Response from mapping operations.TrNrUrr,)r-r.r/r0r1rrr2rUrrr3rr4r,r5r#rr:s0+GT!%E8DI %E8C=r5rc\rSrSr%SrSr\\\S'\ R"SSS9r \\ \S'Sr \\\S'Sr\\\S 'Sr\\\S 'Sr\\\S 'Sr\\\S 'Sr\\\S 'Sr\\\S'Srg)V1ExtractParamsi@z0Parameters for extracting information from URLs.Nr+rorIrprrenableWebSearchrorigin showSourcesrr,)r-r.r/r0r1r+rr3r2rLrMrprrrrrrrrrrsr4r,r5r#rr@s: FHSM "*..X"FL(3-F"&L(3-&)--&*OXd^*(,x~, FHSM "&K$&/3M8O,3r5rc\rSrSr%SrSr\\\S'Sr \\ S\S'Sr \\ \S'Sr \\S 'Sr\\\S 'Sr\\\S 'Sr\\\S 'Sr\\\\S 'Sr\\\S'Sr\\\\4\S'Sr\\\S'Srg)V1ExtractResponseiLz!Response from extract operations.Nr processingrfrgrrTrrrrwarnings replacementsourcesrr,)r-r.r/r0r1rrr3r2rr rr rrrr%rrr#rr$r%rrrrr4r,r5r#r r Ls+B EIFHW@A BI$(Ix!(GTD(1+E8C=!GXc]!$(HhtCy!(!%K#%(,GXd38n %,!%K#%r5r c\rSrSr%\\S'Sr\\\S'Sr \\\S'Sr \\\S'Sr \\\S 'S r \\\S 'Sr \\\S 'S r\\\S'Sr\\\S'Sr\\\S'Srg)V1SearchParamsiZqueryrNtbsfilterenlangusr`rapiri`rrr,)r-r.r/r0r3r2rrrr*r+r-r`rrrrrsr4r,r5r#r'r'Zs JE8C=C# FHSM D(3-!GXc]!"Hhsm"!FHSM!"GXc]"/3M8O,3r5r'c`\rSrSr%SrSr\\S'\\ \S'Sr \ \ \S'Sr \ \ \S'S rg) V1SearchResponseifz Response from search operations.TrrNrrr,)r-r.r/r0r1rrr2rrOrrr3rr4r,r5r#r1r1fs7*GT " ##!GXc]!E8C=r5r1c\\rSrSr%\\S'Sr\\\S'Sr\\ \S'Sr \\ \S'Sr g)V1CreditUsageDataimremaining_creditsN plan_creditsbilling_period_startbilling_period_endr,) r-r.r/r0rr2r5rr6r3r7r4r,r5r#r3r3ms4"&L(3-&*.(3-.(, ,r5r3c*\rSrSr%\\S'\\S'Srg)V1CreditUsageResponseisrrr,N)r-r.r/r0rr2r3r4r,r5r#r9r9ss M r5r9c\\rSrSr%\\S'Sr\\\S'Sr\\ \S'Sr \\ \S'Sr g)V1TokenUsageDataiwremaining_tokensN plan_tokensr6r7r,) r-r.r/r0rr2r=rr6r3r7r4r,r5r#r;r;ws4!%K#%*.(3-.(, ,r5r;c*\rSrSr%\\S'\\S'Srg)V1TokenUsageResponsei}rrr,N)r-r.r/r0rr2r;r4r,r5r#r?r?}s M r5r?c\\rSrSr%Sr\\\S'Sr\\\S'Sr \\\S'\ \S'Sr g)V1CreditUsageHistoricalPeriodiN startDateendDateapiKeyrr, r-r.r/r0rBrr3r2rCrDrr4r,r5r#rArAs2#Ix}#!GXc]! FHSM r5rAc0\rSrSr%\\S'\\\S'Srg)V1CreditUsageHistoricalResponseirperiodsr,N) r-r.r/r0rr2rrAr4r,r5r#rGrGs M / 00r5rGc\\rSrSr%Sr\\\S'Sr\\\S'Sr \\\S'\ \S'Sr g)V1TokenUsageHistoricalPeriodiNrBrCrD tokensUsedr,rEr,r5r#rJrJs1#Ix}#!GXc]! FHSM Or5rJc0\rSrSr%\\S'\\\S'Srg)V1TokenUsageHistoricalResponseirrHr,N) r-r.r/r0rr2rrJr4r,r5r#rMrMs M . //r5rMcj\rSrSr%SrSr\\\S'Sr \\ \S'Sr \\ \S'S r \\ \S 'S r g ) V1GenerateLLMsTextParamsiz3 Parameters for the LLMs.txt generation operation. maxUrlsF showFullTextTcacheN._V1GenerateLLMsTextParams__experimental_streamr,)r-r.r/r0r1rQrrr2rRrrSrTr4r,r5r#rOrOsB GXc]#(L(4.( E8D> ,08D>0r5rOc\rSrSr%SrSr\\\S'Sr \\\S'Sr \\\S'S r \\ \S 'S r \\ \S 'S r\\\S 'S rg )V1DeepResearchParamsiz- Parameters for the deep research operation. ri timeLimitrQNanalysisPromptr/_V1DeepResearchParams__experimental_streamStepsr,)r-r.r/r0r1rrrr2rXrQrZr3rr[rr4r,r5r#rVrVs^ Hhsm"Ix}"GXc]$(NHSM("&L(3-&155r5rVcp\rSrSr%Sr\\S'\\S'Sr\ \\S'Sr \ \ \\S'Sr \ \\S'S r g) V1DeepResearchResponseiz, Response from the deep research operation. rrNrr#r$r,)r-r.r/r0r1rr2r3rrr#rr$r4r,r5r#r]r]sBM GE8C=$(HhtCy!(!%K#%r5r]c\rSrSr%Sr\\S'Sr\\ \ \ 4\S'\ \S'Sr \\ \S'\ \S'\ \S '\ \S '\\ \ \ 4\S '\\ \ \ 4\S '\\ \S 'Sr\\\ \S'Sr\\ \S'Srg)V1DeepResearchStatusResponseiz3 Status response from the deep research operation. rNrrrr currentDepthr activitiesr% summariesr#r$r,)r-r.r/r0r1rr2rrrr3rrrrr#r$r4r,r5r#r_r_sM%)D(4S> ") KE8C=NMT#s(^$$ $sCx. !!Cy$(HhtCy!(!%K#%r5r_ct\rSrSr%SrSr\\S'\\S'Sr \ \\S'Sr \ \ \\S'Sr \ \\S 'S rg) V1GenerateLLMsTextResponseiz-Response from LLMs.txt generation operations.TrrNrr#r$r,)r-r.r/r0r1rrr2r3rrr#rr$r4r,r5r#rdrdsC7GT GE8C=$(HhtCy!(!%K#%r5rdc4\rSrSr%\\S'Sr\\\S'Srg)$V1GenerateLLMsTextStatusResponseDataillmstxtN llmsfulltxtr,) r-r.r/r0r3r2rhrr4r,r5r#rfrfs L!%K#%r5rfc\rSrSr%SrSr\\S'Sr\ \ \S'\ S\S'Sr \ \ \S '\ \S 'Sr\ \\ \S 'Sr\ \ \S 'S rg) V1GenerateLLMsTextStatusResponseiz4Status response from LLMs.txt generation operations.TrNrr!rrrr#r$r,)r-r.r/r0r1rrr2rrrfr rr3r#rr$r4r,r5r#rjrjs_>GT;?D(7 8? 7 88E8C=N$(HhtCy!(!%K#%r5rjcf\rSrSr%Sr\\S'\\\ \ 4\S'Sr \ \ \S'Sr \ \ \S'Srg) r1iz% Response from the search operation. rrNrrr,)r-r.r/r0r1rr2rrr3rrrrr4r,r5r#r1r1s=M tCH~ !GXc]!E8C=r5c\rSrSr%SrSr\\\S'\ R"SSS9r \\ \S'Sr \\\S'S r\\\S 'S r\\\S 'S r\\\S 'S r\\\S 'Sr\\\\ 4\S'Srg)riz' Parameters for the extract operation. Nr+rorIrp system_promptFallow_external_linksenable_web_searchr show_sourcesrr,)r-r.r/r0r1r+rr3r2rLrMrprrmrnrrorrprrr4r,r5r#rrs!FHSM "*..X"FL(3-F#'M8C='+0(4.0(-x~-&+OXd^+#(L(4.(&*E8DcN #*r5c0R\rSrSrSrSrSS\\S\\SS4SjjrSSSSSSS SSSSSSSSSSSSSSS .S \S \\ \ S S\\ \\4S\\ \S\\ \S\\ S\\ S\\ S\\S\\ S\\ S\\ S\\ S\\ SS\\ S\\S\\S\\ \\\\\\\\\\4 S\\S \\ S!\\ S"\\ S\\4.S#jjrSSSSSSS SS$.S%\S&\\ S'\\S(\\S)\\S*\\S\\S\\ S+\\S\4S,jjr S\!4S-jr"S\#4S.jr$SS0\ S\%4S1jjr&SS0\ S\'4S2jjr(SSSSSSSSSSSSSSSSSSS3SS4.S \S5\\ \S6\\ \S7\\ S8\\ S&\\ S9\\ S:\\ S;\\ S<\\ S+\\S=\\\\)4S>\\ S?\\ S@\\ SA\\ SB\\ SC\\ S"\\ SD\\ SE\\S\*4,SFjjr+SSSSSSSSSSSSSSSSSSSSG.S \S5\\ \S6\\ \S7\\ S8\\ S&\\ S9\\ S:\\ S;\\ S<\\ S+\\S=\\\\)4S>\\ S?\\ S@\\ SA\\ SB\\ SC\\ S"\\ SE\\S\,4*SHjjr-SI\S\*4SJjr.SI\S\/4SKjr0SI\S\ \\44SLjr1SSSSSSSSSSSSSSSSSSSSG.S \S5\\ \S6\\ \S7\\ S8\\ S&\\ S9\\ S:\\ S;\\ S<\\ S+\\S=\\\\)4S>\\ S?\\ S@\\ SA\\ SB\\ SC\\ S"\\ SE\\SSM4*SNjjr2SSSSSS SSSO.S \SP\\S<\\ SQ\\ SR\\ S&\\ S\\ SS\\ S\\S\34STjjr4SSSSSSS SSSSSSSSSSS3SSSSU.SV\ \S \\ \ SWS\\ \\4S\\ \S\\ \S\\ S\\ S\\ S\\S\\ S\\ S\\ S\\ S\\ SS\\S\\S\\ \\\\\\\\\\4 SX\\5SD\\ SC\\ S"\\ SE\\S\64.SYjjr7SSSSSSS SSSSSSSSSSSSSSZ.SV\ \S \\ \ SWS\\ \\4S\\ \S\\ \S\\ S\\ S\\ S\\S\\ S\\ S\\ S\\ S\\ SS\\S\\S\\ \\\\\\\\\\4 SX\\5SC\\ SE\\S"\\ S\84,S[jjr9SSSSSSS SSSSSSSSSSSSSS\.SV\ \S \\ \ SWS\\ \\4S\\ \S\\ \S\\ S\\ S\\ S\\S\\ S\\ S\\ S\\ S\\ SS\\S\\S\\ \\\\\\\\\\4 SX\\5SC\\ S"\\ SE\\SSM4,S]jjr:SI\S\64S^jr;SI\S\/4S_jr<SSSSS/S/S/SS`.SV\\ \Sa\\Sb\\Sc\\S;\\ Sd\\ Se\\ SX\\ \\4S\=\4Sfjjjr>Sg\S\=\4Shjr?SSSSS/S/S/SS`.SV\\ \Sa\\Sb\\Sc\\S;\\ Sd\\ Se\\ SX\\ \\4S\=\4Sijjjr@SSSSSj.S \Sk\\ Sl\\ Sm\\ Sn\\ S\A4 SojjrBSSSSSj.S \Sk\\ Sl\\ Sm\\ Sn\\ S\C4 SpjjrDSI\S\A4SqjrESSE\\S\ \\44SrjjrFSS \Ss\ \\4S\ \\4St\ Su\GS\HR4 SvjjrJSS \S\ \\4St\ Su\GS\HR4 SwjjrKSS \S\ \\4St\ Su\GS\HR4 SxjjrLSI\S\ \\4SD\ S\*4SyjrMSz\HRS{\SS4S|jrNS}\ S{\S~\S\S\4 SjrOSSSSSSSSS.S%\S7\\ S\\ Sk\\ S\\Sc\\S\\ S\\P\ \\4/S4S\\P\ \\4/S4S\Q4SjjrRSSSSSSS.S%\S7\\ S\\ Sk\\ S\\Sc\\S\\ S\ \\44SjjrSSI\S\Q4SjrTS\ \\4S\SS4SjrUSrVSSjrWSrXSSjrYSrZSr[Sr\SSjr]SSjr^Sr_g)V1FirecrawlAppia Legacy v1 Firecrawl client for backward compatibility. This class provides the complete v1 API implementation including: - URL scraping with various formats and options - Website crawling with monitoring capabilities - Batch scraping operations - Search functionality - Data extraction with LLM integration - Deep research capabilities - LLMs.txt generation This is used by the unified client to provide version-specific access through app.v1.method_name() patterns. aSchema contains invalid structure for OpenAI: object type with no 'properties' defined but 'additionalProperties: true' (schema-less dictionary not supported by OpenAI). Please define specific properties for your object. Note: Recursive schemas using '$ref' are supported.Napi_keyapi_urlreturncTU=(d [R"S5UlU=(d [R"SS5UlSUR;a-URc [R S5 [ S5e[RSUR35 g) z Initialize the V1FirecrawlApp instance with API key, API URL. Args: api_key (Optional[str]): API key for authenticating with the Firecrawl API. api_url (Optional[str]): Base URL for the Firecrawl API. FIRECRAWL_API_KEYFIRECRAWL_API_URLhttps://api.firecrawl.devzapi.firecrawl.devNz%No API key provided for cloud servicezNo API key providedz)Initialized V1FirecrawlApp with API URL: )rgetenvrsrtrr ValueErrordebug)selfrsrts r#__init__V1FirecrawlApp.__init__ s{@")),?"@ ]")),?A\"]  $,, .4<<3G NNB C23 3 @OPr5r~)ryre include_tags exclude_tagsonly_main_contentwait_forrrrskip_tls_verificationremove_base64_images block_adsr parse_pdfrV json_optionsrYchange_tracking_optionsmax_agestore_in_cachezero_data_retentionrQryrvrerrrrrrrrrrrrrrVrrYrrrrc URUS5 UR5nUS[3S.nU(aUUS'U(aUUS'U(aUUS'U(aUUS'UbUUS 'U(aUUS 'U(aUUS 'U (aU RS S S 9US'U bU US'U bU US'U bU US'U bU US'U(aUUS'UbUUS'UbmUR U5n[ U[5(aSU;aUR US5US'[ U[5(aUOURS S S 9US'UbmUR U5n[ U[5(aSU;aUR US5US'[ U[5(aUOURS S S 9US'U(a;UVs/sH*n[ U[5(aUOURS S S 9PM, snUS'U(a*[ U[5(aUOURS S S 9US'UbUUS'UbUUS'UbUUS'UR U5 SU;a0US(a&SUS;aUR USS5USS'SU;a0US(a&SUS;aUR USS5USS'[R"URS3UUUbUS- S-OSS 9nURS!:Xa`UR5nURS"5(aS#U;a[S(0US#D6$S$U;a[S%US$35e[S%U35eUR!US'5 gs snf![a [S&5ef=f))ag Scrape and extract content from a URL. Args: url (str): Target URL to scrape formats (Optional[List[Literal["markdown", "html", "rawHtml", "content", "links", "screenshot", "screenshot@fullPage", "extract", "json"]]]): Content types to retrieve (markdown/html/etc) headers (Optional[Dict[str, str]]): Custom HTTP headers include_tags (Optional[List[str]]): HTML tags to include exclude_tags (Optional[List[str]]): HTML tags to exclude only_main_content (Optional[bool]): Extract main content only wait_for (Optional[int]): Wait for a specific element to appear timeout (Optional[int]): Request timeout (ms) location (Optional[LocationConfig]): Location configuration mobile (Optional[bool]): Use mobile user agent skip_tls_verification (Optional[bool]): Skip TLS verification remove_base64_images (Optional[bool]): Remove base64 images block_ads (Optional[bool]): Block ads proxy (Optional[Literal["basic", "stealth", "auto"]]): Proxy type (basic/stealth) extract (Optional[JsonConfig]): Content extraction settings json_options (Optional[JsonConfig]): JSON extraction settings actions (Optional[List[Union[WaitAction, ScreenshotAction, ClickAction, WriteAction, PressAction, ScrollAction, ScrapeAction, ExecuteJavascriptAction, PDFAction]]]): Actions to perform change_tracking_options (Optional[ChangeTrackingOptions]): Change tracking settings zero_data_retention (Optional[bool]): Whether to delete data after scrape is done Returns: ScrapeResponse with: * Requested content formats * Page metadata * Extraction results * Success/error status Raises: Exception: If scraping fails scrape_url python-sdk@rQrryrerzr{Nr|r}rTby_alias exclude_nonerrrrrrrrorVrrYrrrzeroDataRetention /v1/scrape@@r)rerHrrrrFailed to scrape URL. Error: +Failed to parse Firecrawl response as JSON.z scrape URLr,)_validate_kwargs_prepare_headersversiondict_ensure_schema_dict isinstanceupdaterequestspostrt status_coderHgetrrr{ _handle_error)r}rQryrerrrrrrrrrrrrrVrrYrrrrkwargs_headers scrape_paramsactionresponse response_jsons r#rV1FirecrawlApp.scrape_urlsY| fl3((*#G9- '.M) $ '.M) $ +7M- ( +7M- (  (/@M+ , '/M) $ '.M) $ (0 tRV (WM* %  &,M( # ,3HM/ 0  +2FM. /  (1M* % %*M' "  (1M* %  ..w7G'4((X-@$($<$?    3 & O (   $$Y//Fm4K+?WaCSgij$$S3   3 & O (   $$Y//I4N:K]KK -#&TUbcjUkTl$mnn#&TUbTc$dee   x)F G O MNN O6B=%B==CcUR5nURS3U(aSOS-nURX25nURS:Xa]UR 5nUR S5(aSU;a [ S 0UD6$SU;a[SUS35e[SU35eURUS 5 g ![a [S 5ef=f) z Get historical token usage (v1).z/v1/team/token-usage/historicalrrrrrHrz-Failed to get historical token usage. Error: rzget token usage historicalNr,) rrtrrrHrrMrr{rrs r#get_token_usage_historical)V1FirecrawlApp.get_token_usage_historicalQs((*=>V`BRfhi$$S3   3 & O (   $$Y//I4N9JMJJ -#&STabiTjSk$lmm#&STaSb$cdd   x)E F O MNN Or) include_paths exclude_paths max_depthmax_discovery_depthrallow_backward_linkscrawl_entire_domainrnignore_sitemaprrdeduplicate_similar_urlsignore_query_parametersregex_on_full_urlrallow_subdomainsmax_concurrencyr poll_intervalidempotency_keyrrrrrrrnrrrrrrrrrrc JURUS5 0nUbUUS'UbUUS'UbUUS'UbUUS'UbUUS'UbUUS'OUbUUS 'U bU US 'U bU US 'U bU RS S S 9US'U bU US'U bU US'UbUUS'UbUUS'UbUUS'UbUUS'UbUUS'UbUUS'URU5 URS5n[ S 0UD6nURS S S 9nUUS'S[ 3US'U(aUUS'UR U5nURURS3UU5nURS:Xa3UR5RS5nURUUU5$URUS5 g! [S5e=f)!a Crawl a website starting from a URL. Args: url (str): Target URL to start crawling from include_paths (Optional[List[str]]): Patterns of URLs to include exclude_paths (Optional[List[str]]): Patterns of URLs to exclude max_depth (Optional[int]): Maximum crawl depth max_discovery_depth (Optional[int]): Maximum depth for finding new URLs limit (Optional[int]): Maximum pages to crawl allow_backward_links (Optional[bool]): DEPRECATED: Use crawl_entire_domain instead crawl_entire_domain (Optional[bool]): Follow parent directory links allow_external_links (Optional[bool]): Follow external domain links ignore_sitemap (Optional[bool]): Skip sitemap.xml processing scrape_options (Optional[ScrapeOptions]): Page scraping configuration webhook (Optional[Union[str, WebhookConfig]]): Notification webhook settings deduplicate_similar_urls (Optional[bool]): Remove similar URLs ignore_query_parameters (Optional[bool]): Ignore URL parameters regex_on_full_url (Optional[bool]): Apply regex to full URLs delay (Optional[int]): Delay in seconds between scrapes allow_subdomains (Optional[bool]): Follow subdomains max_concurrency (Optional[int]): Maximum number of concurrent scrapes zero_data_retention (Optional[bool]): Whether to delete data after 24 hours poll_interval (Optional[int]): Seconds between status checks (default: 2) idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: CrawlStatusResponse with: * Crawling status and progress * Crawled page contents * Success/error information Raises: Exception: If crawl fails crawl_urlNrrrrrrrrrTrrrrrrrrrrrrQrr /v1/crawlrrrstart crawl jobr,)rrrrrrr _post_requestrtrrHr_monitor_job_statusr)r}rQrrrrrrrrnrrrrrrrrrrrrr crawl_paramsrrrrerrs r#rV1FirecrawlApp.crawl_urlesX~ fk2   $+8L (  $+8L (  '0L $  *0CL, -  $)L !  *0CL, - ! -1EL- .  +1EL- .  %,:L )  %,:,?,?\`,?,aL )  &-L # # /5ML1 2 " .4KL0 1  (->L) *  $)L !  '.>L* +  &-L) *  $)L !  '.>L* +  &-a P"MOO$`&/2]&_ _ )""LL+LQC)PQ!"s=G4AG?9G/ #G? G,/ G<<G?? H, H''H,cUR5nURURSUS3U5nURS:Xa[ S0UR 5D6$URUS5 g! [ S5e=f)a Returns information about crawl errors. Args: id (str): The ID of the crawl job Returns: V1CrawlErrorsResponse containing: * errors (List[Dict[str, str]]): List of errors with fields: - id (str): Error ID - timestamp (str): When the error occurred - url (str): URL that caused the error - error (str): Error message * robotsBlocked (List[str]): List of URLs blocked by robots.txt Raises: Exception: If error check fails r/errorsrrzcheck crawl errorsNr,rrrtrrrHrrr}rrers r#check_crawl_errors!V1FirecrawlApp.check_crawl_errorss&'')$$ ~Zt7%KWU   3 & P,?x}}??   x)= > P"MOO A00 A=cUR5nURURSU3U5nURS:XaUR 5$UR US5 g! [ S5e=f)- Cancel an asynchronous crawl job. Args: id (str): The ID of the crawl job to cancel Returns: Dict[str, Any] containing: * success (bool): Whether cancellation was successful * error (str, optional): Error message if cancellation failed Raises: Exception: If cancellation fails rrrzcancel crawl jobN)r_delete_requestrtrrHrrrs r# cancel_crawlV1FirecrawlApp.cancel_crawlsz'')''4<<. 2$(GQ   3 & P}}&   x); < P"MOOs A&& A3V1CrawlWatcherc UR"U40SU_SU_SU_SU_SU_SU_SU_SU _S U _S U _S U _S U _S U_SU_SU_SU_SU_SU_SU_UD6nUR(a'UR(a[URU5$[ S5e)am Initiate a crawl job and return a CrawlWatcher to monitor the job via WebSocket. Args: url (str): Target URL to start crawling from include_paths (Optional[List[str]]): Patterns of URLs to include exclude_paths (Optional[List[str]]): Patterns of URLs to exclude max_depth (Optional[int]): Maximum crawl depth max_discovery_depth (Optional[int]): Maximum depth for finding new URLs limit (Optional[int]): Maximum pages to crawl allow_backward_links (Optional[bool]): DEPRECATED: Use crawl_entire_domain instead crawl_entire_domain (Optional[bool]): Follow parent directory links allow_external_links (Optional[bool]): Follow external domain links ignore_sitemap (Optional[bool]): Skip sitemap.xml processing scrape_options (Optional[V1ScrapeOptions]): Page scraping configuration webhook (Optional[Union[str, V1WebhookConfig]]): Notification webhook settings deduplicate_similar_urls (Optional[bool]): Remove similar URLs ignore_query_parameters (Optional[bool]): Ignore URL parameters regex_on_full_url (Optional[bool]): Apply regex to full URLs delay (Optional[int]): Delay in seconds between scrapes allow_subdomains (Optional[bool]): Follow subdomains max_concurrency (Optional[int]): Maximum number of concurrent scrapes zero_data_retention (Optional[bool]): Whether to delete data after 24 hours idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: V1CrawlWatcher: An instance to monitor the crawl job via WebSocket Raises: Exception: If crawl job fails to start rrrrrrrrnrrrrrrrrrrrCrawl job failed to start)rrrrr)r}rQrrrrrrrrnrrrrrrrrrrrrcrawl_responses r#crawl_url_and_watch"V1FirecrawlApp.crawl_url_and_watchs r--  ' (   !4    "6 !4 "6 * *  &> %< 0 ! ".# $,% &!4' (,+ .  ! !n&7&7!."3"3T: :78 8r5)rrinclude_subdomains sitemap_onlyrr use_indexrrrrrc URU S5 0n UbX+S'UbX;S'UbXKS'UbX[S'UbXkS'UbX{S'UbXS 'U bU RS S S 9U S 'U RU 5 U RS 5n [ S0U D6n U RS S S 9nXS'S[ 3US'U (aXS '[ R"URS3SSUR30US9nURS:Xa]UR5nURS5(aSU;a [S0UD6$SU;a[SUS35e[SU35eURUS5 g![a [S5ef=f)a Map and discover links from a URL. Args: url (str): Target URL to map search (Optional[str]): Filter pattern for URLs ignore_sitemap (Optional[bool]): Skip sitemap.xml processing include_subdomains (Optional[bool]): Include subdomain links sitemap_only (Optional[bool]): Only use sitemap.xml limit (Optional[int]): Maximum URLs to return timeout (Optional[int]): Request timeout in milliseconds **kwargs: Additional parameters to pass to the API Returns: V1MapResponse: Response containing: * success (bool): Whether request succeeded * links (List[str]): Discovered URLs * error (Optional[str]): Error message if any Raises: Exception: If mapping fails or response cannot be parsed map_urlNrrrrrrrTrrrrQrr/v1/maprrrrrrUrzMap failed. Error: rmapr,)rrrrrrrrrtrsrrHrrr{r)r}rQrrrrrrrrr map_paramsrrrrrs r#rV1FirecrawlApp.map_urlAsH fi0   #)x  %*8 '  ).@* +  #(4} %  "'w   $+y !  %.z "  %-]]Dt]%TJz " &!!~~m4 #0Z0 "''D'I  E"-gY 7 H )5 &==||nG $$ ~&>?    3 & O (   $$Y//G}4L(9=99 -#&9-:P9Q$RSS#&9-$IJJ   x / O MNN Os76E&.%E&&E<)ryrerrrrrrrrrrrrVrrYrrrrrurls rRrSrTrwrUrWrxrVrHrc 0URUS5 0nUbUUS'UbUUS'UbUUS'UbUUS'UbUUS'UbUUS'UbUUS 'U bU RS S S 9US 'U bU US 'U bU US'U bU US'U bU US'UbUUS'UbmURU5n[U[5(aSU;aURUS5US'[U[5(aUOURS S S 9US'UbmURU5n[U[5(aSU;aURUS5US'[U[5(aUOURS S S 9US'U(a;UVs/sH*n[U[5(aUOURS S S 9PM, snUS'UbURS S S 9US'UbUUS'UbUUS'UR U5 [ S!0UD6nURS S S 9nUUS'S[ 3US'SU;a0US(a&SUS;aURUSS5USS'SU;a0US(a&SUS;aURUSS5USS'URU5nURURS3UU5nURS:Xa3UR5RS5nURUUU5$URUS 5 gs snf! [S5e=f)"aH Batch scrape multiple URLs and monitor until completion. Args: urls (List[str]): URLs to scrape formats (Optional[List[Literal]]): Content formats to retrieve headers (Optional[Dict[str, str]]): Custom HTTP headers include_tags (Optional[List[str]]): HTML tags to include exclude_tags (Optional[List[str]]): HTML tags to exclude only_main_content (Optional[bool]): Extract main content only wait_for (Optional[int]): Wait time in milliseconds timeout (Optional[int]): Request timeout in milliseconds location (Optional[LocationConfig]): Location configuration mobile (Optional[bool]): Use mobile user agent skip_tls_verification (Optional[bool]): Skip TLS verification remove_base64_images (Optional[bool]): Remove base64 encoded images block_ads (Optional[bool]): Block advertisements proxy (Optional[Literal]): Proxy type to use extract (Optional[JsonConfig]): Content extraction config json_options (Optional[JsonConfig]): JSON extraction config actions (Optional[List[Union]]): Actions to perform agent (Optional[AgentOptions]): Agent configuration max_concurrency (Optional[int]): Maximum number of concurrent scrapes poll_interval (Optional[int]): Seconds between status checks (default: 2) idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: V1BatchScrapeStatusResponse with: * Scraping status and progress * Scraped content for each URL * Success/error information Raises: Exception: If batch scrape fails batch_scrape_urlsNryrerzr{r|r}rTrrrrrrrrorVrrYrrrrrr/v1/batch/scraperrrstart batch scrape jobr,)rrrrrrrrrrtrrHrrrr)r}rryrerrrrrrrrrrrrVrrYrrrrrrrrrrrrs r#r V1FirecrawlApp.batch_scrape_urlss@ f&9:   '.M) $  '.M) $  #+7M- (  #+7M- (  (/@M+ ,  '/M) $  '.M) $  (0 tRV (WM* %  &,M( # ,3HM/ 0  +2FM. /  (1M* %  %*M' "  ..w7G'4((X-@$($<$8{[dOeCe/3/G/G T]H^_gHh/iK "8 , K 'K ,F8WbcpWqKq373K3KKXeLfgoLp3qK &x 0''8%%6F&GV]^   3 & P]]_((.++BG G   x)A BC(U: P"MOOs61L>L L)ryrerrrrrrrrrrrrVrrYrrrrc URUS5 0nUbUUS'UbUUS'UbUUS'UbUUS'UbUUS'UbUUS'UbUUS 'U bU RS S S 9US 'U bU US 'U bU US'U bU US'U bU US'UbUUS'UbmURU5n[U[5(aSU;aURUS5US'[U[5(aUOURS S S 9US'UbmURU5n[U[5(aSU;aURUS5US'[U[5(aUOURS S S 9US'U(a;UVs/sH*n[U[5(aUOURS S S 9PM, snUS'UbURS S S 9US'UbUUS'UbUUS'UR U5 [ S 0UD6nURS S S 9nUUS'S[ 3US'SU;a0US(a&SUS;aURUSS5USS'SU;a0US(a&SUS;aURUSS5USS'URU5nURURS3UU5nURS:Xa[S 0UR5D6$URUS5 gs snf! [S5e=f)!av Initiate a batch scrape job asynchronously. Args: urls (List[str]): URLs to scrape formats (Optional[List[Literal]]): Content formats to retrieve headers (Optional[Dict[str, str]]): Custom HTTP headers include_tags (Optional[List[str]]): HTML tags to include exclude_tags (Optional[List[str]]): HTML tags to exclude only_main_content (Optional[bool]): Extract main content only wait_for (Optional[int]): Wait time in milliseconds timeout (Optional[int]): Request timeout in milliseconds location (Optional[LocationConfig]): Location configuration mobile (Optional[bool]): Use mobile user agent skip_tls_verification (Optional[bool]): Skip TLS verification remove_base64_images (Optional[bool]): Remove base64 encoded images block_ads (Optional[bool]): Block advertisements proxy (Optional[Literal]): Proxy type to use extract (Optional[JsonConfig]): Content extraction config json_options (Optional[JsonConfig]): JSON extraction config actions (Optional[List[Union]]): Actions to perform agent (Optional[AgentOptions]): Agent configuration max_concurrency (Optional[int]): Maximum number of concurrent scrapes zero_data_retention (Optional[bool]): Whether to delete data after 24 hours idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: V1BatchScrapeResponse with: * success - Whether job started successfully * id - Unique identifier for the job * url - Status check URL * error - Error message if start failed Raises: Exception: If job initiation fails async_batch_scrape_urlsNryrerzr{r|r}rTrrrrrrrrorVrrYrrrrrrrrrrr,)rrrrrrrrrrtrrrHrr)r}rryrerrrrrrrrrrrrVrrYrrrrrrrrrrs r#r&V1FirecrawlApp.async_batch_scrape_urls+s@ f&?@   '.M) $  '.M) $  #+7M- (  #+7M- (  (/@M+ ,  '/M) $  '.M) $  (0 tRV (WM* %  &,M( # ,3HM/ 0  +2FM. /  (1M* %  %*M' "  ..w7G'4((X-@$($<$8{[dOeCe/3/G/G T]H^_gHh/iK "8 , K 'K ,F8WbcpWqKq373K3KKXeLfgoLp3qK &x 0''8%%6F&GV]^   3 & P,?x}}??   x)A BA(U: P"MOOs61K*>K// K<)ryrerrrrrrrrrrrrVrrYrrrrc URUS5 0nUbUUS'UbUUS'UbUUS'UbUUS'UbUUS'UbUUS'UbUUS 'U bU RS S S 9US 'U bU US 'U bU US'U bU US'U bU US'UbUUS'UbmURU5n[U[5(aSU;aURUS5US'[U[5(aUOURS S S 9US'UbmURU5n[U[5(aSU;aURUS5US'[U[5(aUOURS S S 9US'U(a;UVs/sH*n[U[5(aUOURS S S 9PM, snUS'UbURS S S 9US'UbUUS'UbUUS'UR U5 [ S!0UD6nURS S S 9nUUS'S[ 3US'SU;a0US(a&SUS;aURUSS5USS'SU;a0US(a&SUS;aURUSS5USS'URU5nURURS3UU5nURS:Xa][S!0UR5D6nUR(a'UR(a[URU5$[!S5eUR#US 5 gs snf! [!S5e=f)"a8 Initiate a batch scrape job and return a CrawlWatcher to monitor the job via WebSocket. Args: urls (List[str]): URLs to scrape formats (Optional[List[Literal]]): Content formats to retrieve headers (Optional[Dict[str, str]]): Custom HTTP headers include_tags (Optional[List[str]]): HTML tags to include exclude_tags (Optional[List[str]]): HTML tags to exclude only_main_content (Optional[bool]): Extract main content only wait_for (Optional[int]): Wait time in milliseconds timeout (Optional[int]): Request timeout in milliseconds location (Optional[LocationConfig]): Location configuration mobile (Optional[bool]): Use mobile user agent skip_tls_verification (Optional[bool]): Skip TLS verification remove_base64_images (Optional[bool]): Remove base64 encoded images block_ads (Optional[bool]): Block advertisements proxy (Optional[Literal]): Proxy type to use extract (Optional[JsonConfig]): Content extraction config json_options (Optional[JsonConfig]): JSON extraction config actions (Optional[List[Union]]): Actions to perform agent (Optional[AgentOptions]): Agent configuration max_concurrency (Optional[int]): Maximum number of concurrent scrapes zero_data_retention (Optional[bool]): Whether to delete data after 24 hours idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: V1CrawlWatcher: An instance to monitor the batch scrape job via WebSocket Raises: Exception: If batch scrape job fails to start batch_scrape_urls_and_watchNryrerzr{r|r}rTrrrrrrrrorVrrYrrrrrrrr Batch scrape job failed to startrrr,)rrrrrrrrrrtrrrHrrrrr)r}rryrerrrrrrrrrrrrVrrYrrrrrrrrrrr s r#r"*V1FirecrawlApp.batch_scrape_urls_and_watchs x f&CD   '.M) $  '.M) $  #+7M- (  #+7M- (  (/@M+ ,  '/M) $  '.M) $  (0 tRV (WM* %  &,M( # ,3HM/ 0  +2FM. /  (1M* %  %*M' "  ..w7G'4((X-@$($<$8{[dOeCe/3/G/G T]H^_gHh/iK "8 , K 'K ,F8WbcpWqKq373K3KKXeLfgoLp3qK &x 0''8%%6F&GV]^   3 & P!6!I!I!))n.?.?).*;*;TBB#$FGG   x)A BI(UB P"MOOs61L->AL2 L22 L?c`SU3nUR5nURURU3U5nURS:XGaUR 5nUSS:XaSU;aUSnSU;a[ US5S:XaOURS5nU(d[RS 5 OURXs5nURS:wa#[RS UR35 O<UR 5n URU RS/55 U nSU;aMXeS'[S0S U;aSOSURS5URS5URS5URS5URS5URS5URS5URS 5S. D6$URUS5 g ! [ S5e=f! [ S5e=f![ a#n [RS U 35 S n A MS n A ff=f)a Check the status of a batch scrape job using the Firecrawl API. Args: id (str): The ID of the batch scrape job. Returns: V1BatchScrapeStatusResponse: The status of the batch scrape job. Raises: Exception: If the status check request fails. /v1/batch/scrape/rrrrfrrrrrrNrFTrrr) rrrrfrrrrrzcheck batch scrape statusr,)rrrtrrHrrrrrrrrrrs r#check_batch_scrape_status(V1FirecrawlApp.check_batch_scrape_statusCs('rd+'')$$ ~hZ%@'J   3 & P&mmo 8$ 3[(&v.D K/{623q8!#.??6#:'"NN+LM! ".2.?.?.RO.::cA & /J?KfKfJg-h i %`,;,@,@,B !KK fb(AB*3K#!K/*+/'. $+{$:5%//(3$1(__[9*}=(__[9#/#/$1 2     x)D EQ P"MOO$`&/2]&_ _ )""LL+LQC)PQ!"s=G 4AH9G0 #H G-0 G==H H- H((H-cUR5nURURSUS3U5nURS:Xa[ S0UR 5D6$URUS5 g! [ S5e=f)a Returns information about batch scrape errors. Args: id (str): The ID of the crawl job. Returns: V1CrawlErrorsResponse containing: * errors (List[Dict[str, str]]): List of errors with fields: * id (str): Error ID * timestamp (str): When the error occurred * url (str): URL that caused the error * error (str): Error message * robotsBlocked (List[str]): List of URLs blocked by robots.txt Raises: Exception: If the error check request fails r&rrrzcheck batch scrape errorsNr,rrs r#check_batch_scrape_errors(V1FirecrawlApp.check_batch_scrape_errorss&'')$$ ~5Frd'%RT[\   3 & P,?x}}??   x)D E P"MOOrr+rormrnrorprr+rormrorpc SSKn U RS[SS9 URU S5 UR 5n U(dU(d [ S5eU(dU(d [ S5eU(aUR U5nU=(d /UUUUS [53S .n U(aX,S 'U(aXLS 'U(aXS 'U RU 5 URURS3U U 5n U RS:XaU R5nUS(aURS5nU(d [S5eURURSU3U 5nURS:XaFUR5nUSS:Xa [!S!0UD6$USS;a[SUSSUS35eOUR#US5 [$R&"S5 M[SUS35eUR#U S5 [!SSS 9$! [S5e=f! [S5e=f![an[ [)U5S5eSnAff=f)"a Extract structured information from URLs. .. deprecated:: The extract endpoint is in maintenance mode and its use is discouraged. Review https://docs.firecrawl.dev/developer-guides/usage-guides/choosing-the-data-extractor to find a replacement. Args: urls (Optional[List[str]]): URLs to extract from prompt (Optional[str]): Custom extraction prompt schema (Optional[Any]): JSON schema/Pydantic model system_prompt (Optional[str]): System context allow_external_links (Optional[bool]): Follow external links enable_web_search (Optional[bool]): Enable web search show_sources (Optional[bool]): Include source URLs agent (Optional[Dict[str, Any]]): Agent configuration **kwargs: Additional parameters to pass to the API Returns: V1ExtractResponse[Any] with: * success (bool): Whether request succeeded * data (Optional[Any]): Extracted data matching schema * error (Optional[str]): Error message if any Raises: ValueError: If prompt/schema missing or extraction fails rNThe extract endpoint is in maintenance mode and its use is discouraged. Review https://docs.firecrawl.dev/developer-guides/usage-guides/choosing-the-data-extractor to find a replacement.r stacklevelrV#Either prompt or schema is required!Either urls or prompt is requiredrrrrrrorr+rr /v1/extractrrrr)Job ID not returned from extract request. /v1/extract/rrfrgr Extract job . Error: rzextract-statusFailed to extract. Error: FzInternal server error.rrr,)r#warnDeprecationWarningrrr{rr$rrrtrrHrrrr rtimesleepr3)r}rr+rormrnrorprrr#re request_datarrjob_idrrrs r#rVV1FirecrawlApp.extractsP   %    fi0'')fBC CF@A A --f5FJB"60'#KM?3   %+ " +8 ( $) ! F#) *))<<. ,H ##s*T#==?D ?!XXd^F!'(STT*.*;*;#||nLA#++66#=`.=.B.B.D  +84 C'8'G;'G G!,X!6:Q!Q&/,{8?T>UU^_jkr_s^t0u&v v"R!..@PQ 1 #&$&@g$PQQ""8Y7!6NOOCT#&QSS`&/2]&_ _ *SVS) ) *sP0H4HAH47H$H4A.H4 H!!H4$ H11H44 I>IIrBcSSKnURS[SS9 UR5nUR UR SU3U5nUR S:Xa[S 0UR5D6$URUS 5 g! [S5e=f![an[[U5S 5eSnAff=f) a Retrieve the status of an extract job. .. deprecated:: The extract endpoint is in maintenance mode and its use is discouraged. Review https://docs.firecrawl.dev/developer-guides/usage-guides/choosing-the-data-extractor to find a replacement. Args: job_id (str): The ID of the extract job. Returns: ExtractResponse[Any]: The status of the extract job. Raises: ValueError: If there is an error retrieving the status. rNr.rr/r6rrzget extract statusr;r,) r#r=r>rrrtrr rHrrr{r3)r}rBr#rerrs r#get_extract_status!V1FirecrawlApp.get_extract_statuss$   %   '') *((DLL>fX)NPWXH##s*T,?x}}??""8-ABT#&QSS *SVS) ) *s/1BB 6B BB B=#B88B=cSSKn U RS[SS9 UR5n UnU(aUR U5nUUUUUS[ 3S.n U(aX+S'U(aXKS 'U(aXS 'UR URS 3X5n U RS :Xa[S0U R5D6$URU S5 g! [S 5e=f![an [[U 5S5eSn A ff=f)a  Initiate an asynchronous extract job. .. deprecated:: The extract endpoint is in maintenance mode and its use is discouraged. Review https://docs.firecrawl.dev/developer-guides/usage-guides/choosing-the-data-extractor to find a replacement. Args: urls (List[str]): URLs to extract information from prompt (Optional[str]): Custom extraction prompt schema (Optional[Any]): JSON schema/Pydantic model system_prompt (Optional[str]): System context allow_external_links (Optional[bool]): Follow external links enable_web_search (Optional[bool]): Enable web search show_sources (Optional[bool]): Include source URLs agent (Optional[Dict[str, Any]]): Agent configuration idempotency_key (Optional[str]): Unique key to prevent duplicate requests Returns: ExtractResponse[Any] with: * success (bool): Whether request succeeded * data (Optional[Any]): Extracted data matching schema * error (Optional[str]): Error message if any Raises: ValueError: If job initiation fails rNr.rr/rr3r+rrr4rrz async extractr;r,)r#r=r>rrrrrtrr rHrrr{r3)r}rr+rormrnrorprr#rerArrs r# async_extractV1FirecrawlApp.async_extractEs%N   %   '') --f5F"60'#G9-   %+ " +8 ( $) ! *))T\\N+*F ^H##s*T,?x}}??""8_=T#&QSS *SVS) ) *s06/C"&C?C" CC"" D,DDmax_urlsshow_full_textrSexperimental_streamrKrLrSrMcSSKnURS[SS9 [UUUUS9nUR UUUUUS9nUR (aUR (d [SS S S S 9$UR n URU 5n U RS :XaU $U RS :XaU $U RS:wa [SSS S S 9$[R"S5 Mi)a Generate LLMs.txt for a given URL and poll until completion. .. deprecated:: /v1/llmstxt is deprecated and will not be replaced. Args: url (str): Target URL to generate LLMs.txt from max_urls (Optional[int]): Maximum URLs to process (default: 10) show_full_text (Optional[bool]): Include full text in output (default: False) cache (Optional[bool]): Whether to use cached content if available (default: True) experimental_stream (Optional[bool]): Enable experimental streaming Returns: GenerateLLMsTextStatusResponse with: * Generated LLMs.txt content * Full version if requested * Generation status * Success/error information Raises: Exception: If generation fails rN3/v1/llmstxt is deprecated and will not be replaced.rr/rQrRrS__experimental_streamrJFz#Failed to start LLMs.txt generationrgrrrrrrfr"/LLMs.txt generation job terminated unexpectedly) r#r=r>rOasync_generate_llms_textrrrjcheck_generate_llms_text_statusrr?r@) r}rQrKrLrSrMr#paramsrrBrs r#generate_llms_text!V1FirecrawlApp.generate_llms_texts>  A   *'"5  00 ) 3 1 x{{3;  99&AF}} + (* ,.7!K#  JJqMr5c&SSKnURS[SS9 [UUUUS9nUR 5nSU0UR SSS 9En S [ 3U S 'URURS 3X5n U R5n [S U 5 [SU 5 U RS5(a [S0U D6$URU S5 [SSS9$! [S5e=f![an [[!U 55eSn A ff=f)aZ Initiate an asynchronous LLMs.txt generation operation. .. deprecated:: /v1/llmstxt is deprecated and will not be replaced. Args: url (str): The target URL to generate LLMs.txt from. Must be a valid HTTP/HTTPS URL. max_urls (Optional[int]): Maximum URLs to process (default: 10) show_full_text (Optional[bool]): Include full text in output (default: False) cache (Optional[bool]): Whether to use cached content if available (default: True) experimental_stream (Optional[bool]): Enable experimental streaming Returns: GenerateLLMsTextResponse: A response containing: * success (bool): Whether the generation initiation was successful * id (str): The unique identifier for the generation job * error (str, optional): Error message if initiation failed Raises: Exception: If the generation job initiation fails. rNrOrr/rPrQTrrr /v1/llmstxt json_datarrrzstart LLMs.txt generationFInternal server errorr<r,)r#r=r>rOrrrrrtrHrrrdrrr{r3) r}rQrKrLrSrMr#rVrer[reqrrs r#rT'V1FirecrawlApp.async_generate_llms_texts6<  A   *'"5  '')CQ6;;4;#PQ +G95 ( %$$ ~[%A9VCxxzH +y ) *h '||I&&S5AAA""8-HI*)  S#$QRR %SV$ $ %s1AC-5 CC- C**C-- D7D  DcSSKnURS[SS9 UR5nUR UR SU3U5nUR S:XaUR5n[S0UD6$UR S :Xa [S 5eURUS 5 [S S SSS9$![an[S[U535eSnAff=f![an[[U55eSnAff=f)a  Check the status of a LLMs.txt generation operation. .. deprecated:: /v1/llmstxt is deprecated and will not be replaced. Args: id (str): The unique identifier of the LLMs.txt generation job to check status for. Returns: GenerateLLMsTextStatusResponse: A response containing: * success (bool): Whether the generation was successful * status (str): Status of generation ("processing", "completed", "failed") * data (Dict[str, str], optional): Generated text with fields: * llmstxt (str): Generated LLMs.txt content * llmsfulltxt (str, optional): Full version if requested * error (str, optional): Error message if generation failed * expiresAt (str): When the generated data expires Raises: Exception: If the status check fails. rNrOrr/ /v1/llmstxt/rzFFailed to parse Firecrawl response as GenerateLLMsTextStatusResponse: z!LLMs.txt generation job not foundz check LLMs.txt generation statusFr\rgrrRr,) r#r=r>rrrtrrHrjrr3rr{)r}rr#rerr[rs r#rU.V1FirecrawlApp.check_generate_llms_text_status' s.  A   '') %((DLL>bT)JGTH##s*w ( I;HiHH%%, CDD""8-OP0E\emy{||!w#&lmpqrmslt$uvvw  %SV$ $ %s;1CB28-C2 C<CCC C>%C99C>cZU(aSSUR3US.$SSUR3S.$)z Prepare the headers for API requests. Args: idempotency_key (Optional[str]): A unique key to ensure idempotency of requests. Returns: Dict[str, str]: The headers including content type, authorization, and optionally idempotency key. zapplication/jsonr) Content-Typerzx-idempotency-key)rdr)rs)r}rs r#rV1FirecrawlApp._prepare_headersV sB  2#*4<<.!9%4 /&t||n5  r5rretriesbackoff_factorc[U5H`n[R"XUSU;aUSb USS- S-OSS9nURS:Xa[R "USU--5 M^Us $ W$)a Make a POST request with retries. Args: url (str): The URL to send the POST request to. data (Dict[str, Any]): The JSON data to include in the POST request. headers (Dict[str, str]): The headers to include in the POST request. retries (int): Number of retries for the request. backoff_factor (float): Backoff factor for retries. Returns: requests.Response: The response from the POST request. Raises: requests.RequestException: If the request fails after the specified retries. rNrr)rr)rangerrrr?r@)r}rQrrerfrgattemptrs r#rV1FirecrawlApp._post_requestn s.W~G}}SqzCrCHLMVHWHcPTU^P_bhPhklPlimoH##s* >Q'\:; & r5c[U5HGn[R"XS9nURS:Xa[R "USU--5 MEUs $ W$)a Make a GET request with retries. Args: url (str): The URL to send the GET request to. headers (Dict[str, str]): The headers to include in the GET request. retries (int): Number of retries for the request. backoff_factor (float): Backoff factor for retries. Returns: requests.Response: The response from the GET request. Raises: requests.RequestException: If the request fails after the specified retries. rerir)rjrrrr?r@r}rQrerfrgrkrs r#rV1FirecrawlApp._get_request sO*W~G||C9H##s* >Q'\:; & r5c[U5HGn[R"XS9nURS:Xa[R "USU--5 MEUs $ W$)a Make a DELETE request with retries. Args: url (str): The URL to send the DELETE request to. headers (Dict[str, str]): The headers to include in the DELETE request. retries (int): Number of retries for the request. backoff_factor (float): Backoff factor for retries. Returns: requests.Response: The response from the DELETE request. Raises: requests.RequestException: If the request fails after the specified retries. rnrir)rjrdeleterr?r@ros r#rV1FirecrawlApp._delete_request sO*W~GsQ'\:; & r5cURSU3nURXB5nURS:XaUR5nUSS:XaSU;a|USnSU;ab[ US5S:XaOOURUSU5nUR5nUR URS/55 SU;aMbXvS'[S0UD6$[ S 5eUSS ;a#[US 5n[R"U5 O#[ S US35eURUS 5 GM+! [ S5e=f! [ S5e=f)a Monitor the status of a crawl job until completion. Args: id (str): The ID of the crawl job. headers (Dict[str, str]): The headers to include in the status check requests. poll_interval (int): Seconds between status checks. Returns: CrawlStatusResponse: The crawl results if the job is completed successfully. Raises: Exception: If the job fails or an error occurs during status checks. rrrrrfrrrz,Crawl job completed but no data was returnedactivepausedpendingqueuedwaitingrrz)Crawl job failed or was stopped. Status: rr,) rtrrrHrrrrr maxr?r@r)r}rrerrtrrrs r#r"V1FirecrawlApp._monitor_job_status s|&j5G"//AO**c1T"1"6"6"8Kx(K7,*62$ 3";v#671< %.2.?.? F@SU\.]O`.=.B.B.D !KK (CD% 3/3F+4C{CC'(VWW *.nn"%mA"6MJJ}-#&OP[\dPeOf$ghh""?4HI?T#&QSS`&/2]&_ _sD-D=- D:= E rrcUR5nURSS5nURSS5nUR URX$U5n[RRXqS 9e! URSSnUR5(aSU3nSUR3nNwS UR3nS nN![ a S UR3nS nNf=f=f) a0 Handle errors from API responses. Args: response (requests.Response): The response object from the API request. action (str): Description of the action that was being performed. Raises: Exception: An exception with a message containing the status code and error details from the response. rNo error message provided.details%No additional error details provided.Nr;z#Server returned non-JSON response: zFull response status: z+Server returned empty response with status zNo additional details availablez0Server returned unreadable response with status )r) rHrrrrr{_get_error_messager exceptions HTTPError)r}rrr error_message error_details response_textmessages r#rV1FirecrawlApp._handle_error s B$MMOM)--g7STM)--i9`aM))(*>*>Wde!!++G+GG# B B ( ds 3  &&((&I-$YM&rVasync_deep_researchrcheck_deep_research_statusrrr?r@)r}r(rrrKrrmrrrr#research_paramsrrBlast_activity_countlast_source_countrnew_activitiesactivity new_sourcessources r# deep_researchV1FirecrawlApp.deep_research: sV  F     *3J '  !+5K (  )1I &  &0?, -  $.;N + & 2O$44V&?#Y&0$Y/0A0BC )Ff%*$'y(9$:!h;. !X-"? 7@S?T UVV!\1 JJqM-0!+VWWr5)rrrKrrmrcSSKnURS[SS9 0n UbX)S'UbX9S'UbXIS'UbXYS 'UbXiS 'UbXyS '[S0U D6n UR 5n S U0U R S S S9En S[ 3U S'SU ;a?U Sn U (a3SU ;a-[U SS5(aU SR5U SS'URURS3X5n U RS:XaU R5$URU S5 SSS.$! [S5e=f![an[[!U55eSnAff=f) Initiates an asynchronous deep research operation. .. deprecated:: /v1/deep-research is deprecated. Use /v2/search instead. Args: query (str): Research query or topic to investigate max_depth (Optional[int]): Maximum depth of research exploration time_limit (Optional[int]): Time limit in seconds for research max_urls (Optional[int]): Maximum number of URLs to process analysis_prompt (Optional[str]): Custom prompt for analysis system_prompt (Optional[str]): Custom system prompt __experimental_stream_steps (Optional[bool]): Enable experimental streaming Returns: Dict[str, Any]: A response containing: * success (bool): Whether the research initiation was successful * id (str): The unique identifier for the research job * error (str, optional): Error message if initiation failed Raises: Exception: If the research initiation fails. rNrrr/rrXrQrZrrr(Trrrrro/v1/deep-researchrrzstart deep researchFr\r<r,)r#r=r>rVrrrhasattrrorrtrrHrrr{r3)r}r(rrrKrrmrr#rrer[ json_optsrrs r#r"V1FirecrawlApp.async_deep_research sD  F     *3J '  !+5K (  )1I &  &0?, -  $.;N + & 2x5H5O5O5Q -(2 %))T\\N:K*LiaH##s*S#==?*""8-BC!+BCCS#$QRR %SV$ $ %s0/D/7DD/ D,,D// E9E  EcSSKnURS[SS9 UR5nUR UR SU3U5nUR S:XaUR5$UR S :Xa [S 5eURUS 5 S S S.$! [S5e=f![an[[U55eSnAff=f) Check the status of a deep research operation. .. deprecated:: /v1/deep-research is deprecated. Use /v2/search instead. Args: id (str): The ID of the deep research operation. Returns: DeepResearchResponse containing: Status: * success - Whether research completed successfully * status - Current state (processing/completed/failed) * error - Error message if failed Results: * id - Unique identifier for the research job * data - Research findings and analysis * sources - List of discovered sources * activities - Research progress log * summaries - Generated research summaries Raises: Exception: If the status check fails. rNrrr//v1/deep-research/rrrazDeep research job not foundzcheck deep research statusFr\r<) r#r=r>rrrtrrHrrr{r3)r}rr#rerrs r#r)V1FirecrawlApp.check_deep_research_status s8  F   '') %((DLL>9KB4)PRYZH##s*S#==?*%%, =>>""8-IJ!+BCCS#$QRR  %SV$ $ %s/1B0B --B0 B--B00 C:CCr method_namec U(dg1Sk1Sk1Sk1Sk1Sk1Sk1Sk1SkS.nURU[55n[UR55U- nU(a![S US S R U5S 35eg) ah Validate additional keyword arguments before they are passed to the API. This provides early validation before the Pydantic model validation. Args: kwargs (Dict[str, Any]): Additional keyword arguments to validate method_name (str): Name of the method these kwargs are for Raises: ValueError: If kwargs contain invalid or unsupported parameters N>rrrYrVryrrrrrrrrrrrrr> r*r-rr+r`rrrr>rrrrrrrrrrrrnrr>rrrrrrr>rr+rorrprmrorn>rrrrYrVryrerrrrrrrrrrr)rrrrrVrrr"zUnsupported parameter(s) for z: z, zC. Please refer to the API documentation for the correct parameters.)rsetkeysr{r)r}rr method_paramsallowed_paramsunknown_paramss r#rV1FirecrawlApp._validate_kwargs& s K|}R"@(F,J% 2'**;>V[[]+n< <[MDIIVdLeKfgjkl l r5cUcU$[U[5(aB[US5(aUR5$[US5(aUR 5$[U[ 5(a4UR 5VVs0sHup#X RU5_M snn$[U[[45(a!UVs/sHo0RU5PM sn$U$s snnfs snf)zg Utility to ensure a schema is a dict, not a Pydantic model class. Recursively checks dicts and lists. model_json_schemaro) rrrrroritemsrlisttuple)r}rokvs r#r"V1FirecrawlApp._ensure_schema_dictY s >M fd # #v233//11**}}& fd # #?E||~N~tqA//22~N N ftUm , ,9?@A,,Q/@ @ O@s C$C*cU(a[U[[45(dgUc [5nSSKnUR "US[ S9nXd;agURU5 [U[5(aSU;a[US[ 5(a}USRS5n[U5S:aZUSS :XaQUS S :XaHUS nX:XaURU5 gX;a&URX8X#U5URU5 $UR5H-n URXX45(dM URU5 g OH[U[5(a3UH-n URXX45(dM URU5 g URU5 g!URU5 f=f) aU Check if an object contains a recursive reference to a specific definition. Args: obj: Object to check target_def_name: Name of the definition to check for recursion defs: Dictionary of definitions visited: Set of visited object keys to detect cycles Returns: True if recursive reference is found, False otherwise FNrT) sort_keysdefault$ref/#r$defs) rrrrrHdumpsr3addsplitrdiscard_contains_recursive_refvalues) r}objtarget_def_namedefsvisitedrHobj_keyref_pathdef_namevalueitems r#r&V1FirecrawlApp._contains_recursive_refk s*S4,77 ?eG**SD#>   G %#t$$S=ZF S%A%A"6{005H8})hqkS.@Xa[T[E[#+B<#6#' OOG $$+#'#?#?P_gn#o OOG $!ZZ\E33EDZZ# OOG $*C&&D33D4YY# OOG $  OOG $ OOG $s1#A2F/(F/(F/?F/2F/F/F//GcxU(dgUR5Hup#URX2U5(dM g g)z Check if $defs contain circular references. Args: defs: Dictionary of definitions to check Returns: True if circular references are found, False otherwise FT)rr)r}rr def_values r#_check_for_circular_defs'V1FirecrawlApp._check_for_circular_defs s8#'::< H++IFF$0r5c U(a![U[[45(aUS:aU$Uc [5n[ U5nXS;aU$UR U5 [U[5(aSU;a[US[ 5(aUSRS5n[U5S:aNUSS:XaEUSS:Xa<US nXr;a2UR[X'5X#US-5URU5 $UURU5 $0nUR5H$upU S:XaM URXX4S-5X'M& UURU5 $[U[5(a6UV s/sHoRXX4S-5PM sn URU5 $URU5 U$s sn f!URU5 f=f) a$ Resolve $ref references in a JSON schema object. Args: obj: Object to resolve references in defs: Dictionary of definitions visited: Set to track visited objects and prevent infinite recursion depth: Current recursion depth Returns: Object with resolved references rPrrrrrrrr) rrrrrrr3rr _resolve_refsrr) r}rrrdepthobj_idrrresolvedrrrs r#rV1FirecrawlApp._resolve_refs s*S4,7752:J ?eGC  J F $#t$$S=ZF S%A%A"6{005H8})hqkS.@Xa[T[E[#+B<#+#'#5#5d4>6JD[`cd[d#e OOF # OOF #"%))+JCg~ $($6$6uGUVY$WHM#. OOF # C&&WZ[WZt**4w JWZ[ OOF # ' OOF # \ OOF #s1BGG;G G9F?G?GGct^^U(a[U[5(dU$[5mUU4SjmT"U5$)z Normalize a schema for OpenAI compatibility by handling recursive references. Args: schema: Schema to normalize Returns: Normalized schema c @>[U[5(d0[U[5(aUVs/sH nT "U5PM sn$U$[U5nUT;aU$TR U5 [U5nSU;aTR U5 U$SU;aUR S5n0nUR5H0upg[U[5(aSU;a T "U5XV'M,XuU'M2 0UESUR5VV s0sHupUT "U 5_M sn n0En TR U5 U $URS5S:XaSU;aURS5SLaUS URS5S:XanSU;ahSU;ab[US[5(aG[US[5(a/USV s/sHn XS;dM U PM n n U (aXS'OUS OUS [UR55Huupg[U[5(aSU;a T "U5X6'M,[U[5(dMCUVs/sH"n[U[5(aT "U5OUPM$ snX6'Mw TR U5 U$s snfs sn nfs sn fs snf) Nrrrobject propertiesadditionalPropertiesTrequired) rrrrrrpoprr)rrr normalizedrprocessed_restrrrrresultfieldvalid_requirednormalize_objectrs r#rEV1FirecrawlApp._normalize_schema_for_openai..normalize_object sc4((c4((?BCst,T2sCC WF  KK cJ#'!!*$!~~g.!#","2"2"4JC"5$//F%4G.>u.E+.3s+ #5 hNgGY]YcYcYe5fYeQUQRa9I!9L6LYe5fg' v&(2 *56$>56v&(2j( *z*5t<<z,7>>9CJ9O&I9O&+,/G&G',9ON&I%1?:.&z2":.#:#3#3#56 eT**vU/B&6u&=JOt,,ns&tnsfjDRVAWAW'7'=]a'ans&tJO 7 OOF # wD46g&I'usJ J ; J J)Jrrr)r}rorrs @@r#_normalize_schema_for_openai+V1FirecrawlApp._normalize_schema_for_openai s3Z55M%> @ ''r5c|^^U(a[U[5(dg[5mUU4SjmT"U5(+$)z Validate schema for OpenAI compatibility. Args: schema: Schema to validate Returns: True if schema is valid, False otherwise Tc>[U[5(dg[U5nUT;agTRU5 SU;aTR U5 gUR S5S:Xa2SU;a,SU;a&UR S5SLaTR U5 gUR 5Hn[U[5(a(SU;a"T"U5(aTR U5 gM@[U[5(dMWUHCn[U[5(dMSU;dM"T"U5(dM1TR U5 g M TR U5 g) NFrrrrpatternPropertiesrT)rrrrrrrr)rrrrhas_invalid_structurers r#rIV1FirecrawlApp._validate_schema_for_openai..has_invalid_structureB s'c4((WF  KK }'8+C'#3../47'eT**vU/B,U33/#4 t,, %%dD11fD6H4T:: ' 7'+ !&& OOF #r5r)r}rorrs @@r#_validate_schema_for_openai*V1FirecrawlApp._validate_schema_for_openai3 s4Z55%# J)000r5cU(a[U[5(dgSSKnUR"U5nSU;=(d SU;=(d SU;n[ UR S5=(d UR S55nU=(d U$) z Detect if a schema contains recursive references. Args: schema: Schema to analyze Returns: True if schema has recursive patterns, False otherwise FrN"$ref"#/$defs/z#/definitions/r definitions)rrrHrrr)r}rorH schema_stringhas_refshas_defss r#_detect_recursive_schema'V1FirecrawlApp._detect_recursive_schemai s|Z55 6*  % . - ' .  -   7+Hvzz-/HI#8#r5cZU(dSSS.$URU5(aSSS.$SSS.$)z Select appropriate model based on schema complexity. Args: schema: Schema to analyze Returns: Dict with modelName and reason z gpt-4o-mini no_schema) modelNamereasonzgpt-4orecursive_schema_detected simple_schema)r)r}ros r#_select_model_for_schema'V1FirecrawlApp._select_model_for_schema s:!.+F F  ( ( 0 0!)5PQ Q*oFFr5c&[U[5(GaZX!;GaTURX5nU(a[U[5(aURS05nSSKnUR "U5nURS5=(d SU;=(d SU;nU(a]UR X45nUR "U5n SU ;=(d SU ;n U (d Un[U[5(a SU;aUS O0UR X45n[U[5(a SU;aUS URU5nURU5(d[UR5eX1U'U$![a NNf=f![a N^f=f)z Process and validate a schema container (like extract or json_options). Args: schema_container: Dict containing schema schema_key: Key where schema is stored (default: "schema") Returns: Processed schema container rrNrr) rrrrrHrrrrrr{OPENAI_SCHEMA_ERROR_MESSAGE) r}schema_container schema_keyrorrHr has_any_refsresolved_schemaresolved_stringhas_remaining_refss r#_process_schema_with_validation.V1FirecrawlApp._process_schema_with_validation s & - -*2P--.>.JKF*VT22zz'2. $ 6 2 JJw'0 -0-/   *.*<*F33F;; !A!ABB+1Z (#%%s%%AE3/F3 F?F FFrsrtNN)FNr?)Nr)ro)`r-r.r/r0r1rrr3r~rr rrrr^rrrrrrrrrrrrlrrrrsr1rr9rr?rrGrrMrrcr rrrrrrrr rrr'rrrrr"r'r*r rVrErHrjrWrdrTrUrrrResponserrrrrrrr_rrrrrrrrrrrrrr4r,r5r#rrrrs$ q Q Qx}QX\Q,mq04040404&*%*37%)4837(,OS(,.237EIIM%)-1261U7U7d7,g$hij U7 d38n- U7 #49- U7#49-U7 (~U7smU7c]U7/0U7TNU7$,D>U7#+4.U7 ~U7 G$JKL!U7" ~#U7$l+%U7&#<0'U7(d57I=ZgivyGIWYrt*$@AB)U7*&..E%F+U7,c]-U7.%TN/U70"*$1U72*#.3U7v$(!%$("&%)&*%*8<]3]3C= ]3 # ]3 SM ]33-]3c]]3sm]3c]]3%_5]3*]3~="7=,#@<$}%@<&#4.'@<("#)@<*&d^+@<, }-@<."#/@<2 3@#z<$}%z<&#4.'z<("#)z<*&d^+z<,"#-z<0 1zKC3-KC#KC+,KCKC (~KC'tnKCD>KC  FGH!KC",'#KC$|,%KC&$u\3E}VceruCESUnp{&{ |}~'KC(')KC* }+KC,"#-KC.&d^/KC0"#1KC4 %5KCbW[,0,0,0,0"&!&/3!%04/3$(KO*./3AE*.)-)-.2/JC3iJC$w(Q RST JC $sCx.) JC tCy) JCtCy)JC$D>JC3-JC#JC+,JCJC (~JC'tnJCD>JC  FGH!JC",'#JC$|,%JC&$u\3E}VceruCESUnp{&{ |}~'JC(')JC*"#+JC,"#-JC.&d^/JC2 3JC`W[,0,0,0,0"&!&/3!%04/3$(KO*./3AE*.)-.2)-/JC3iJC$w(Q RST JC $sCx.) JC tCy) JCtCy)JC$D>JC3-JC#JC+,JCJC (~JC'tnJCD>JC  FGH!JC",'#JC$|,%JC&$u\3E}VceruCESUnp{&{ |}~'JC(')JC*"#+JC,&d^-JC."#/JC2 3JCX<FC<F4O<F|FCF4IF>)-~P%)$(+/3805+0.2~P49%~PSM ~P SM ~P $C= ~P#+4.~P (~~P#4.~PDcN+~P+3/~P@%*%*1B31G%*R)-O*%)$(+/3805+0.2O*49%O*SM O* SM O* $C= O*#+4.O* (~O*#4.O*DcN+O*8I7MO*j'+-1$(26LLsm L %TN L D> L"*$L<\Ld'+-1$(26A A sm A %TN A D> A "*$A FJDHgXgX } gX ! gX sm gX&c]gX$C=gX*2$gX"(DcN+;T+A"BCgX $sCx.)94)? @AgXNjgXZ(,(,&*-1+/:>NDND } ND ! ND sm ND&c]ND$C=ND*2$NDDHS>ND`1DS1D5Q1Df.ltCH~.lC.lD.lf$0d&0dO(b41l$0G$6 r5rrc\rSrSrSrS\S\4SjrSSjrSS jr S \S \ \ \\ 4/S4SS4S jr S \S \ \\ 4SS4SjrS\ \\ 4SS4SjrSrg)ri a A class to watch and handle crawl job events via WebSocket connection. Attributes: id (str): The ID of the crawl job to watch app (V1FirecrawlApp): The V1FirecrawlApp instance data (List[Dict[str, Any]]): List of crawled documents/data status (str): Current status of the crawl job ws_url (str): WebSocket URL for the crawl job event_handlers (dict): Dictionary of event type to list of handler functions rappcXlX l/UlSUlURR SS5SU3Ul///S.Ulg)Nrhttpwsr)donerdocument)rrrrrtreplacews_urlevent_handlers)r}rrs r#r~V1CrawlWatcher.__init__ sQ*,   ,,VT:;:bTJ  r5ruNc$# [R"URSSSURR34/S9IShvN nUR U5IShvN SSS5IShvN gN.NN !,IShvN (df  g=f7f)zE Establishes WebSocket connection and starts listening for messages. Nrr)max_sizeadditional_headers websocketsconnectrrrs_listenr} websockets r#r&V1CrawlWatcher.connect s%% KK!0GDHH.S\S?\\\S@\\\SA\\ SB\\ SC\\ SD\\SE\\SF\\SG\\SH\\1SI\\ \\24SJ\\SK\\SL\\SM\\ SN\\S9\\ S\\S \34(SOjjr4SSSSSSSSSSSSSSSSS5SS>.S\S?\\\S@\\\SA\\ SB\\ SC\\ SD\\SE\\SF\\SG\\SH\\1SI\\ \\24SJ\\SK\\SL\\SM\\ SN\\S9\\ S\\S \54(SPjjr6SQ\S \34SRjr7SSQ\S \\\4S9\ S \34SSjjr8SSSSSS SSST.S\SU\\SG\\SV\\SW\\SC\\ S(\\ S)\\S\\9S \:4SXjjr;SSSSSYSYSYSSZ.S\\\S[\\S\\\ S]\\SF\\S^\\S_\\S8\\\\ 4S \<\ 4S`jjjr=SQ\S \-4Sajr>SQ\S \?4Sbjr@SQ\S \?4ScjrASQ\S \\\ 44SdjrBSe\S \<\ 4SfjrCSSSSSYSYSYSSZ.S\\\S[\\S\\\ S]\\SF\\S^\\S_\\S8\\\\ 4S \<\ 4SgjjjrDSSSSh.S\Si\\ Sj\\Sk\\S \E4 SljjrFSSSSSm.S\Si\\ Sj\\Sn\\Sk\\S \G4 SojjrHSQ\S \E4SpjrISSSSSSSSSq.Sr\SA\\ Ss\\ Si\\ St\\S]\\Su\\Sv\\J\\\ 4/S4Sw\\J\\\ 4/S4S \K4SxjjrLSSSSSSSy.Sr\SA\\ Ss\\ Si\\ St\\S]\\Su\\S \\\ 44SzjjrMSQ\S \K4S{jrNSSSSSSS SSS|. Sr\SC\\ S}\\S~\\S\\S\\S)\\S(\\ SH\\1S\\ \\\ 4\O4S \P4SjjrQSrRU=rS$)AsyncV1FirecrawlAppi' z Asynchronous version of V1FirecrawlApp that implements async methods using aiohttp. Provides non-blocking alternatives to all V1FirecrawlApp operations. rsrtc >[TU]XS9 g)Nrsuperr~)r}rsrt __class__s r#r~AsyncV1FirecrawlApp.__init__- s :r5NmethodrQrerrfrgruc B# [R"5IShvN n[U5HnURXX4S9IShvN n U RS:Xa7[ R "USU--5IShvN SSS5IShvN MdU RS:aURU SUS35IShvN U R5IShvN sSSS5IShvN s sSSS5IShvN $ [S 5eNNNNNUN?N1N !,IShvN (df  GM=f![Ra;n XS- :XaU e[ R "USU--5IShvN Sn A GMQSn A ff=f!,IShvN (df  g=f7f) a Generic async request method with exponential backoff retry logic. Args: method (str): The HTTP method to use (e.g., "GET" or "POST"). url (str): The URL to send the request to. headers (Dict[str, str]): Headers to include in the request. data (Optional[Dict[str, Any]]): The JSON data to include in the request body (only for POST requests). retries (int): Maximum number of retry attempts (default: 3). backoff_factor (float): Factor to calculate delay between retries (default: 0.5). Delay will be backoff_factor * (2 ** retry_count). Returns: Dict[str, Any]: The parsed JSON response from the server. Raises: aiohttp.ClientError: If the request fails after all retries. Exception: If max retries are exceeded or other errors occur. N)rQrQrerHriri,zmake z requestrzMax retries exceeded) aiohttp ClientSessionrjrequestrasyncior@rrH ClientErrorr) r}rQrQrerrfrgsessionrkrrs r#_async_request"AsyncV1FirecrawlApp._async_request1 sp6((**g > I&% /  !#??c1")--!w,0O"PPP$     $??c1"&"4"4Xvhh?W"XXX%-]]_4   +**)23 3!+ Q  Y4 +    **IA+-!--!w,(GHHHHI+**s3FDFFD3D D30D8D 9D> D3 D  D3F)D9D :DD D D3!D "D3&F( F4D5F: FF D3 D D3DDD3FD0 D  D0 *D3-F0D33F)E=0E3 1E=6F=FFF F FFcF# URSXX$U5IShvN $N7f)a Make an async POST request with exponential backoff retry logic. Args: url (str): The URL to send the POST request to. data (Dict[str, Any]): The JSON data to include in the request body. headers (Dict[str, str]): Headers to include in the request. retries (int): Maximum number of retry attempts (default: 3). backoff_factor (float): Factor to calculate delay between retries (default: 0.5). Delay will be backoff_factor * (2 ** retry_count). Returns: Dict[str, Any]: The parsed JSON response from the server. Raises: aiohttp.ClientError: If the request fails after all retries. Exception: If max retries are exceeded or other errors occur. POSTNrY)r}rQrrerfrgs r#_async_post_request'AsyncV1FirecrawlApp._async_post_request^ s$*((tn]]]] !!cF# URSXSX45IShvN $N7f)ac Make an async GET request with exponential backoff retry logic. Args: url (str): The URL to send the GET request to. headers (Dict[str, str]): Headers to include in the request. retries (int): Maximum number of retry attempts (default: 3). backoff_factor (float): Factor to calculate delay between retries (default: 0.5). Delay will be backoff_factor * (2 ** retry_count). Returns: Dict[str, Any]: The parsed JSON response from the server. Raises: aiohttp.ClientError: If the request fails after all retries. Exception: If max retries are exceeded or other errors occur. GETNr])r}rQrerfrgs r#_async_get_request&AsyncV1FirecrawlApp._async_get_requestu s$(((dG\\\\r`rrcR# UR5IShvN nURSS5nURSS5nUR URX$U5IShvN n[R"U5eNc! [R"SUR35e=fND7f)a Handle errors from async API responses with detailed error messages. Args: response (aiohttp.ClientResponse): The response object from the failed request action (str): Description of the action that was being attempted Raises: aiohttp.ClientError: With a detailed error message based on the response status: - 402: Payment Required - 408: Request Timeout - 409: Conflict - 500: Internal Server Error - Other: Unexpected error with status code Nrr~rr?Failed to parse Firecrawl error response as JSON. Status code: )rHrrSrWr_get_async_error_messager}rr error_datarrrs r#r!AsyncV1FirecrawlApp._handle_error s  {'}}.J&NN74PQM&NN96]^M55hoov^kll!!'**/ {%%(ghphwhwgx&yz zls8B'A=A;(A= B' B%!B';A==%B""B'rrrc.# URXX45$7fa Generate a standardized error message based on HTTP status code for async operations. Args: status_code (int): The HTTP status code from the response action (str): Description of the action that was being performed error_message (str): The error message from the API response error_details (str): Additional error details from the API response Returns: str: A formatted error message rrs r#rg,AsyncV1FirecrawlApp._get_async_error_message &&{MYYc# UR5nURURS3U5IShvN n[S0UD6$N7f)z8Get current credit usage and billing period (v1, async).rNr,)rrcrtr9r}reresps r#r$AsyncV1FirecrawlApp.get_credit_usage sO''),,||n1 2   %,t,,  3AAAc# UR5nURURS3U5IShvN n[S0UD6$N7f)z7Get current token usage and billing period (v1, async).rNr,)rrcrtr?rrs r#r#AsyncV1FirecrawlApp.get_token_usage sO''),,||n0 1   $+d++  rurVrAsyncV1CrawlWatcherc# URXU5IShvN nURS5(aSU;a[USU5$[S5eN:7f)a Initiate an async crawl job and return an AsyncV1CrawlWatcher to monitor progress via WebSocket. Args: url (str): Target URL to start crawling from params (Optional[V1CrawlParams]): See V1CrawlParams model for configuration: URL Discovery: * includePaths - Patterns of URLs to include * excludePaths - Patterns of URLs to exclude * maxDepth - Maximum crawl depth * maxDiscoveryDepth - Maximum depth for finding new URLs * limit - Maximum pages to crawl Link Following: * allowBackwardLinks - DEPRECATED: Use crawlEntireDomain instead * crawlEntireDomain - Follow parent directory links * allowExternalLinks - Follow external domain links * ignoreSitemap - Skip sitemap.xml processing Advanced: * scrapeOptions - Page scraping configuration * webhook - Notification webhook settings * deduplicateSimilarURLs - Remove similar URLs * ignoreQueryParameters - Ignore URL parameters * regexOnFullURL - Apply regex to full URLs idempotency_key (Optional[str]): Unique key to prevent duplicate requests Returns: AsyncV1CrawlWatcher: An instance to monitor the crawl job via WebSocket Raises: Exception: If crawl job fails to start Nrrr )rrrxr)r}rQrVrr s r#r 'AsyncV1FirecrawlApp.crawl_url_and_watch sYL $33CQQ   i ( (T^-C&~d';TB B78 8 RAA;Arc# URXU5IShvN nURS5(aSU;a[USU5$[S5eN:7f)a Initiate an async batch scrape job and return an AsyncV1CrawlWatcher to monitor progress. Args: urls (List[str]): List of URLs to scrape params (Optional[V1ScrapeParams]): See V1ScrapeParams model for configuration: Content Options: * formats - Content formats to retrieve * includeTags - HTML tags to include * excludeTags - HTML tags to exclude * onlyMainContent - Extract main content only Request Options: * headers - Custom HTTP headers * timeout - Request timeout (ms) * mobile - Use mobile user agent * proxy - Proxy type Extraction Options: * extract - Content extraction config * jsonOptions - JSON extraction config * actions - Actions to perform idempotency_key (Optional[str]): Unique key to prevent duplicate requests Returns: AsyncV1CrawlWatcher: An instance to monitor the batch scrape job via WebSocket Raises: Exception: If batch scrape job fails to start Nrrr#)rrrxr)r}rrVrbatch_responses r#r"/AsyncV1FirecrawlApp.batch_scrape_urls_and_watch sYH $;;D/ZZ   i ( (T^-C&~d';TB B>? ? [r{r~)ryrerrrrrrrrrrrrrVrrYryrvrrrrrrrrrrrrrrVrrYc# URUS5 UR5nUS[3S.nU(aUUS'U(aUUS'U(aUUS'U(aUUS'UbUUS 'U(aUUS 'U(aUUS 'U (aU RS S S 9US'U bU US'U bU US'U bU US'U bU US'U(aUUS'UbUUS'UbmUR U5n[ U[5(aSU;aUR US5US'[ U[5(aUOURS S S 9US'UbmUR U5n[ U[5(aSU;aUR US5US'[ U[5(aUOURS S S 9US'U(a;UVs/sH*n[ U[5(aUOURS S S 9PM, snUS'SU;a0US(a&SUS;aUR USS5USS'SU;a0US(a&SUS;aUR USS5USS'SnUR URU3UU5IShvN nURS5(aSU;a[S0USD6$SU;a[SUS35eURS[U55n[SU35es snfNs7f)a1 Scrape a single URL asynchronously. Args: url (str): Target URL to scrape formats (Optional[List[Literal["markdown", "html", "rawHtml", "content", "links", "screenshot", "screenshot@fullPage", "extract", "json"]]]): Content types to retrieve (markdown/html/etc) headers (Optional[Dict[str, str]]): Custom HTTP headers include_tags (Optional[List[str]]): HTML tags to include exclude_tags (Optional[List[str]]): HTML tags to exclude only_main_content (Optional[bool]): Extract main content only wait_for (Optional[int]): Wait for a specific element to appear timeout (Optional[int]): Request timeout (ms) location (Optional[V1LocationConfig]): Location configuration mobile (Optional[bool]): Use mobile user agent skip_tls_verification (Optional[bool]): Skip TLS verification remove_base64_images (Optional[bool]): Remove base64 images block_ads (Optional[bool]): Block ads proxy (Optional[Literal["basic", "stealth", "auto"]]): Proxy type (basic/stealth) extract (Optional[V1JsonConfig]): Content extraction settings json_options (Optional[V1JsonConfig]): JSON extraction settings actions (Optional[List[Union[V1WaitAction, V1ScreenshotAction, V1ClickAction, V1WriteAction, V1PressAction, V1ScrollAction, V1ScrapeAction, V1ExecuteJavascriptAction, V1PDFAction]]]): Actions to perform **kwargs: Additional parameters to pass to the API Returns: V1ScrapeResponse with: * success - Whether scrape was successful * markdown - Markdown content if requested * html - HTML content if requested * rawHtml - Raw HTML content if requested * links - Extracted links if requested * screenshot - Screenshot if requested * extract - Extracted data if requested * json - JSON data if requested * error - Error message if scrape failed Raises: Exception: If scraping fails rrrryrerzr{Nr|r}rTrrrrrrrrrorVrrYrrrrrr,) rrrrrrr^rtrrrr3)r}rQryrerrrrrrrrrrrrrVrrYrrrrrr error_contents r#rAsyncV1FirecrawlApp.scrape_urlsz fl3((*#G9- '.M) $ '.M) $ +7M- ( +7M- (  (/@M+ , '/M) $ '.M) $ (0 tRV (WM* %  &,M( # ,3HM/ 0  +2FM. /  (1M* % %*M' "  (1M* %  ..w7G'4((X-@$($<$8{[dOeCe/3/G/G T]H^_gHh/iK "8 , K 'K ,F8WbcpWqKq373K3KKXeLfgoLp3qK &x 0''811||n, -     << " " P\\$'77G]SS S   x)A BC(m(  P"MOOSsCE2K;4K"C9K;K'K;#K)4K; K9 K;) K66K;)ryrerrrrrrrrrrrrVrrYrrrrc(# URUS5 0nUbUUS'UbUUS'UbUUS'UbUUS'UbUUS'UbUUS'UbUUS 'U bU RS S S 9US 'U bU US 'U bU US'U bU US'U bU US'UbUUS'UbmURU5n[U[5(aSU;aURUS5US'[U[5(aUOURS S S 9US'UbmURU5n[U[5(aSU;aURUS5US'[U[5(aUOURS S S 9US'U(a;UVs/sH*n[U[5(aUOURS S S 9PM, snUS'UbURS S S 9US'UbUUS'UR U5 [ S 0UD6nURS S S 9nUUS'S[ 3US'SU;a0US(a&SUS;aURUSS5USS'SU;a0US(a&SUS;aURUSS5USS'URU5nURURS3UU5IShvN nURS5S:Xa[S 0UR5D6$URUS5IShvN gs snfNS! [S5e=fN7f)!a, Initiate a batch scrape job asynchronously. Args: urls (List[str]): URLs to scrape formats (Optional[List[Literal]]): Content formats to retrieve headers (Optional[Dict[str, str]]): Custom HTTP headers include_tags (Optional[List[str]]): HTML tags to include exclude_tags (Optional[List[str]]): HTML tags to exclude only_main_content (Optional[bool]): Extract main content only wait_for (Optional[int]): Wait time in milliseconds timeout (Optional[int]): Request timeout in milliseconds location (Optional[LocationConfig]): Location configuration mobile (Optional[bool]): Use mobile user agent skip_tls_verification (Optional[bool]): Skip TLS verification remove_base64_images (Optional[bool]): Remove base64 encoded images block_ads (Optional[bool]): Block advertisements proxy (Optional[Literal]): Proxy type to use extract (Optional[JsonConfig]): Content extraction config json_options (Optional[JsonConfig]): JSON extraction config actions (Optional[List[Union]]): Actions to perform agent (Optional[AgentOptions]): Agent configuration zero_data_retention (Optional[bool]): Whether to delete data after 24 hours idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: V1BatchScrapeResponse with: * success - Whether job started successfully * id - Unique identifier for the job * url - Status check URL * error - Error message if start failed Raises: Exception: If job initiation fails rNryrerzr{r|r}rTrrrrrrrrorVrrYrrrrrrrrrrr,)rrrrrrrrr^rtrrrHrr)r}rryrerrrrrrrrrrrrVrrYrrrrrrrrrs r#r+AsyncV1FirecrawlApp.async_batch_scrape_urls.s| f&?@   '.M) $  '.M) $  #+7M- (  #+7M- (  (/@M+ ,  '/M) $  '.M) $  (0 tRV (WM* %  &,M( # ,3HM/ 0  +2FM. /  (1M* %  %*M' "  ..w7G'4((X-@$($<$8{[dOeCe/3/G/G T]H^_gHh/iK "8 , K 'K ,F8WbcpWqKq373K3KKXeLfgoLp3qK &x 0''811||n, -     << &# - P,?x}}??$$X/GH H HE(U,  P"MOO HsCE6L81K9)DL*K>+LLL3L4 L L  L)rrrrrrrrnrrrrrrrrrrrrrrrrrrnrrrrrrrrc$# URUS5 0nUbUUS'UbUUS'UbUUS'UbUUS'UbUUS'UbUUS'OUbUUS 'U bU US 'U bU US 'U bU RS S S 9US'U bU US'U bU US'UbUUS'UbUUS'UbUUS'UbUUS'URU5 [S0UD6nURS S S 9nUUS'S[3US'UR U5nUR URS3UU5IShvN nURS5(a-URS5nURUUU5IShvN $URUS5IShvN gNb! [S5e=fN1N7f)a  Crawl a website starting from a URL. Args: url (str): Target URL to start crawling from include_paths (Optional[List[str]]): Patterns of URLs to include exclude_paths (Optional[List[str]]): Patterns of URLs to exclude max_depth (Optional[int]): Maximum crawl depth max_discovery_depth (Optional[int]): Maximum depth for finding new URLs limit (Optional[int]): Maximum pages to crawl allow_backward_links (Optional[bool]): DEPRECATED: Use crawl_entire_domain instead crawl_entire_domain (Optional[bool]): Follow parent directory links allow_external_links (Optional[bool]): Follow external domain links ignore_sitemap (Optional[bool]): Skip sitemap.xml processing scrape_options (Optional[V1ScrapeOptions]): Page scraping configuration webhook (Optional[Union[str, V1WebhookConfig]]): Notification webhook settings deduplicate_similar_urls (Optional[bool]): Remove similar URLs ignore_query_parameters (Optional[bool]): Ignore URL parameters regex_on_full_url (Optional[bool]): Apply regex to full URLs delay (Optional[int]): Delay in seconds between scrapes allow_subdomains (Optional[bool]): Follow subdomains poll_interval (Optional[int]): Seconds between status checks (default: 2) idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: V1CrawlStatusResponse with: * Crawling status and progress * Crawled page contents * Success/error information Raises: Exception: If crawl fails rNrrrrrrrrrTrrrrrrrrrQrrrrrrrr,) rrrrrrr^rtrrrr)r}rQrrrrrrrrnrrrrrrrrrrrrrrrerrs r#rAsyncV1FirecrawlApp.crawl_urls3v fk2   $+8L (  $+8L (  '0L $  *0CL, -  $)L !  *0CL, - ! -1EL- .  +1EL- .  %,:L )  %,:,?,?\`,?,aL )  &-L # # /5ML1 2 " .4KL0 1  (->L) *  $)L !  '.>L* + F#%4|4 "''D'I  E"-gY 7 H''811 \\N) $k7<< << " " P\\$'77G]SS S$$X/@A A A<  P"MOOS AsHDFE:F3E<FF F4F5F< F  FFc# 0nUbUUS'UbUUS'UbUUS'UbUUS'UbUUS'UbUUS'OUbUUS'U bU US 'U bU US 'U bU RS S S 9US 'U bU US'U bU US'UbUUS'UbUUS'UbUUS'UbUUS'URU5 [S0UD6nURS S S 9nUUS'S[3US'UR U5nUR UR S3UU5IShvN nURS5(a [S0UD6$URUS5IShvN gNA! [S5e=fN7f)a Start an asynchronous crawl job. Args: url (str): Target URL to start crawling from include_paths (Optional[List[str]]): Patterns of URLs to include exclude_paths (Optional[List[str]]): Patterns of URLs to exclude max_depth (Optional[int]): Maximum crawl depth max_discovery_depth (Optional[int]): Maximum depth for finding new URLs limit (Optional[int]): Maximum pages to crawl allow_backward_links (Optional[bool]): DEPRECATED: Use crawl_entire_domain instead crawl_entire_domain (Optional[bool]): Follow parent directory links allow_external_links (Optional[bool]): Follow external domain links ignore_sitemap (Optional[bool]): Skip sitemap.xml processing scrape_options (Optional[ScrapeOptions]): Page scraping configuration webhook (Optional[Union[str, WebhookConfig]]): Notification webhook settings deduplicate_similar_urls (Optional[bool]): Remove similar URLs ignore_query_parameters (Optional[bool]): Ignore URL parameters regex_on_full_url (Optional[bool]): Apply regex to full URLs idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: V1CrawlResponse with: * success - Whether crawl started successfully * id - Unique identifier for the crawl job * url - Status check URL for the crawl * error - Error message if start failed Raises: Exception: If crawl initiation fails NrrrrrrrrrTrrrrrrrrrQrrrrrrr,) rrrrrr^rtrrrr)r}rQrrrrrrrrnrrrrrrrrrrrrrrrers r#r#AsyncV1FirecrawlApp.async_crawl_url2s p   $+8L (  $+8L (  '0L $  *0CL, -  $)L !  *0CL, - ! -1EL- .  +1EL- .  %,:L )  %,:,?,?\`,?,aL )  &-L # # /5ML1 2 " .4KL0 1  (->L) *  $)L !  '.>L* + F#%4|4 "''D'I  E"-gY 7 H''811 \\N) $     << " " P&222$$X/@A A A  P"MOO As6DEEE! E +EEE EErc B# UR5nSU3nURURU3U5IShvN nURS5S:XaSU;aUSnSU;a[ US5S:XaOrURS5nU(d[ R S5 ODURXb5IShvN nURURS/55 UnSU;aMXTS'[URS5URS 5URS5URS 5URS 5URS5S U;aS OSS9nS U;aURS 5Ul SU;aURS5Ul U$GN_N7f)a Check the status and results of an asynchronous crawl job. Args: id (str): Unique identifier for the crawl job Returns: V1CrawlStatusResponse containing: Status Information: * status - Current state (scraping/completed/failed/cancelled) * completed - Number of pages crawled * total - Total pages to crawl * creditsUsed - API credits consumed * expiresAt - Data expiration timestamp Results: * data - List of crawled documents * next - URL for next page of results (if paginated) * success - Whether status check succeeded * error - Error message if failed Raises: Exception: If status check fails rNrrfrrrrrrrrFT)rrrfrrrr) rrcrtrrrrrr rr r}rrerrrrrrs r#r&AsyncV1FirecrawlApp.check_crawl_statuss2'')t$ 33||nXJ '   ??8 $ 3$"6* +;v./14*v6H#'HI&*&=&=h&P PIKK fb 9:"+K +'+F#(??8,//'*!ook2# 6!ook2($ 3E  k !(__W5HN [ 'OOF3HMI !Qs)9FFA?F;F<-F+B0FFc# URURSU3U5IShvN nURS5S:XaSU;aUSnSU;a[US5S:XaOrURS5nU(d[R S5 ODURXb5IShvN nUR URS/55 UnSU;aMXTS'[S 0UD6$[S 5eURS5S ;a)[R"[US 55IShvN O[S US35eGM@GNNN7f)a Monitor the status of an asynchronous job until completion. Args: id (str): The ID of the job to monitor headers (Dict[str, str]): Headers to include in status check requests poll_interval (int): Seconds between status checks (default: 2) Returns: V1CrawlStatusResponse: The job results if completed successfully Raises: Exception: If the job fails or an error occurs during status checks rNrrfrrrrz&Job completed but no data was returnedrurz#Job failed or was stopped. Status: r,) rcrtrrrrrr rrVr@r{)r}rrerrrrrs r#r-AsyncV1FirecrawlApp._async_monitor_job_statussN $ 7 7<<. 2$/!K x(K7[(&v.D K/{623q8!#.??6#:'"NN+LM!*.*A*A(*T$T  IMM&"$=>&/ !K/+/'0?;??#$LMM*.nnmmC q$9:::"EkRZF[E\ ]^^5%U;s;&E EA?E (E)-E AE *E +E E  E )rrrrrrrrVrrrc<# 0n U (aU RU RSSS95 UbX*S'UbX:S'UbXJS'UbXZS'UbXjS'UbXzS 'UbURSSS9U S '[S0U D6n U RSSS9n XS 'S [3U S 'Sn UR UR U 3U SSUR 30S9IShvN nURS5(aSU;a [S0UD6$SU;a[SUS35e[SU35eNP7f)a Asynchronously map and discover links from a URL. Args: url (str): Target URL to map params (Optional[V1MapParams]): See V1MapParams model: Discovery Options: * search - Filter pattern for URLs * ignoreSitemap - Skip sitemap.xml * includeSubdomains - Include subdomain links * sitemapOnly - Only use sitemap.xml Limits: * limit - Max URLs to return * timeout - Request timeout (ms) Returns: V1MapResponse with: * Discovered URLs * Success/error status Raises: Exception: If mapping fails TrNrrrrrrrrQrrrrrrnrrUrzFailed to map URL. Error: r,) rrrrr^rtrsrrr)r}rQrrrrrrrrVrrrrrs r#rAsyncV1FirecrawlApp.map_urlsH    fkk4dkK L  #)x  %*8 '  ).@* +  #(4} %  "'w   $+y !  %-]]Dt]%TJz "#0Z0 "''D'I  E"-gY 7 H11||nXJ ' $ ~&>?2   << " "w(': ,8, ,  8'9J8KLM M8 CD D sCD D ADFr,r+rormrorpcp# SSKn U RS[SS9 UR5n U(dU(d [ S5eU(dU(d [ S5eU(aUR U5nU=(d /UUUUS[ 53S .n U(aX+S 'U(aXKS 'U(aXS 'URURS 3U U 5IShvN n U RS5(aU RS5n U (d [S5eURURSU 3U 5IShvN nUSS:Xa [S0UD6$USS;a[SUSSUS35e[R"S5IShvN M}[SU RS535eNNwN'7f)a Asynchronously extract structured information from URLs. .. deprecated:: The extract endpoint is in maintenance mode and its use is discouraged. Review https://docs.firecrawl.dev/developer-guides/usage-guides/choosing-the-data-extractor to find a replacement. Args: urls (Optional[List[str]]): URLs to extract from prompt (Optional[str]): Custom extraction prompt schema (Optional[Any]): JSON schema/Pydantic model system_prompt (Optional[str]): System context allow_external_links (Optional[bool]): Follow external links enable_web_search (Optional[bool]): Enable web search show_sources (Optional[bool]): Include source URLs agent (Optional[Dict[str, Any]]): Agent configuration Returns: V1ExtractResponse with: * Structured data matching schema * Source information if requested * Success/error status Raises: ValueError: If prompt/schema missing or extraction fails rNr.rr/r1r2rr3r+rrr4rrr5r6rrfr7r8r9rr:r,)r#r=r>rr{rr$r^rtrrrcr rVr@)r}rr+rormrnrorprr#rerArrBrs r#rVAsyncV1FirecrawlApp.extractasN   %   '')fBC CF@A A --f5FJB"60'#KM?3   %+ " +8 ( $) !11||nK (     << " "\\$'F KLL$($;$;||nL9% x(K7,;{;; *.EE#l;x3H2IS^_fSgRh$ijjmmA&&&8g9N8OPQ Q1 's8CF6F0A"F6:F2;AF6 F4 $F62F64F6c # UR5nSU3nURURU3U5IShvN nUSS:XaSU;aUSnSU;a[US5S:XaOrUR S5nU(d[ R S5 ODURXb5IShvN nURUR S/55 UnSU;aMXTS'[UR S5UR S 5UR S5UR S 5UR S 5UR S5S 9nS U;aUS US 'SU;aUSUS'SS U;aS0UE$S0UE$GN=N7f)a Check the status of an asynchronous batch scrape job. Args: id (str): The ID of the batch scrape job Returns: V1BatchScrapeStatusResponse containing: Status Information: * status - Current state (scraping/completed/failed/cancelled) * completed - Number of URLs scraped * total - Total URLs to scrape * creditsUsed - API credits consumed * expiresAt - Data expiration timestamp Results: * data - List of scraped documents * next - URL for next page of results (if paginated) * success - Whether status check succeeded * error - Error message if failed Raises: Exception: If status check fails r&NrrfrrrrrrrrrrFT) rrcrtrrrrrrrs r#r'-AsyncV1FirecrawlApp.check_batch_scrape_statuss2'')&rd+ 33||nXJ '   x K /$"6* +;v./14*v6H#'HI&*&=&=h&P PIKK fb 9:"+K +'+F#.??8,//'*!ook2# 6!ook2(   k ! +G 4HW  [ *62HV  ; 6u   <@   G !Qs)9E=E8A3E=/E;0-E=BE=;E=c# UR5nURURSUS3U5IShvN $N7f)a Get information about errors from an asynchronous batch scrape job. Args: id (str): The ID of the batch scrape job Returns: V1CrawlErrorsResponse containing: errors (List[Dict[str, str]]): List of errors with fields: * id (str): Error ID * timestamp (str): When the error occurred * url (str): URL that caused the error * error (str): Error message * robotsBlocked (List[str]): List of URLs blocked by robots.txt Raises: Exception: If error check fails r&rNrrcrtr}rres r#r*-AsyncV1FirecrawlApp.check_batch_scrape_errors sJ&''),,||n-bT 9     6?=?c# UR5nURURSUS3U5IShvN $N7f)a Get information about errors from an asynchronous crawl job. Args: id (str): The ID of the crawl job Returns: V1CrawlErrorsResponse containing: * errors (List[Dict[str, str]]): List of errors with fields: - id (str): Error ID - timestamp (str): When the error occurred - url (str): URL that caused the error - error (str): Error message * robotsBlocked (List[str]): List of URLs blocked by robots.txt Raises: Exception: If error check fails rrNrrs r#r&AsyncV1FirecrawlApp.check_crawl_errors$sI&''),,||nJrd' 2     rc# UR5n[R"5IShvN nURURSU3US9IShvN nUR 5IShvN sSSS5IShvN sSSS5IShvN $NfN@N*NN !,IShvN (df  O=fSSS5IShvN g!,IShvN (df  g=f7f)rNrrn)rrSrTrrrtrH)r}rrerXrs r#r AsyncV1FirecrawlApp.cancel_crawl=s'')((**g~~j&Ew~WW[c%]]_,XWW+**W,X+WWW+*****s*C#BC#$C BC B,B-B0 C <B=C  C# BC#C BC C#B4 #B&$ B4 0C 7 C#CC# C C  C C#rBc# SSKnURS[SS9 UR5nUR UR SU3U5IShvN $N![ an[[U55eSnAff=f7f)a Check the status of an asynchronous extraction job. .. deprecated:: The extract endpoint is in maintenance mode and its use is discouraged. Review https://docs.firecrawl.dev/developer-guides/usage-guides/choosing-the-data-extractor to find a replacement. Args: job_id (str): The ID of the extraction job Returns: V1ExtractResponse[Any] with: * success (bool): Whether request succeeded * data (Optional[Any]): Extracted data matching schema * error (Optional[str]): Error message if any * warning (Optional[str]): Warning message if any * sources (Optional[List[str]]): Source URLs if requested Raises: ValueError: If status check fails rNr.rr/r6 r#r=r>rrcrtrr{r3)r}rBr#rers r#rE&AsyncV1FirecrawlApp.get_extract_statusQs.   %   '') %00<<. VH5  %SV$ $ %:*A>$AAAA>A A;"A66A;;A>c # SSKn U RS[SS9 UR5n U(dU(d [ S5eU(dU(d [ S5eU(aUR U5n[ U=(d /UUUUS[3S 9n U(aX+S 'U(aXKS 'U(aXS 'URURS 3U U 5IShvN $N![an [ [U 55eSn A ff=f7f)a. Initiate an asynchronous extraction job without waiting for completion. .. deprecated:: The extract endpoint is in maintenance mode and its use is discouraged. Review https://docs.firecrawl.dev/developer-guides/usage-guides/choosing-the-data-extractor to find a replacement. Args: urls (Optional[List[str]]): URLs to extract from prompt (Optional[str]): Custom extraction prompt schema (Optional[Any]): JSON schema/Pydantic model system_prompt (Optional[str]): System context allow_external_links (Optional[bool]): Follow external links enable_web_search (Optional[bool]): Enable web search show_sources (Optional[bool]): Include source URLs agent (Optional[Dict[str, Any]]): Agent configuration idempotency_key (Optional[str]): Unique key to prevent duplicate requests Returns: V1ExtractResponse[Any] with: * success (bool): Whether request succeeded * data (Optional[Any]): Extracted data matching schema * error (Optional[str]): Error message if any Raises: ValueError: If job initiation fails rNr.rr/r1r2rr3r+rrr4) r#r=r>rr{rr rr^rtrr3) r}rr+rormrnrorprr#rerArs r#rH!AsyncV1FirecrawlApp.async_extractysN   %   '')fBC CF@A A --f5F(3-$  *   %+ " +8 ( $) ! %11<<. ,   %SV$ $ %s<B3D6#C CC DC D*C>>DDrKrLrMrKrLrMc# SSKnURS[SS9 0nUbX&S'UbX6S'UbXFS'URUUUUS 9IShvN nUR S 5(aS U;aU$US nUR U5IShvN n U S S :XaU $U S S:Xa[ SU R S535eU S S:waO [R"S5IShvN Mt[SSSSS9$NNmN7f)a Generate LLMs.txt for a given URL and monitor until completion. .. deprecated:: /v1/llmstxt is deprecated and will not be replaced. Args: url (str): Target URL to generate LLMs.txt from max_urls (Optional[int]): Maximum URLs to process (default: 10) show_full_text (Optional[bool]): Include full text in output (default: False) experimental_stream (Optional[bool]): Enable experimental streaming Returns: V1GenerateLLMsTextStatusResponse containing: * success (bool): Whether generation completed successfully * status (str): Status of generation (processing/completed/failed) * data (Dict[str, str], optional): Generated text with fields: - llmstxt (str): Generated LLMs.txt content - llmsfulltxt (str, optional): Full version if requested * error (str, optional): Error message if generation failed * expiresAt (str): When the generated data expires Raises: Exception: If generation fails rNrOrr/rQrRrQrrrrrfrgz#LLMs.txt generation failed. Error: rr"FrSrrR) r#r=r>rTrrUrrVr@rj) r}rQrKrLrMr#rVrrBrs r#rW&AsyncV1FirecrawlApp.generate_llms_textsQ@  A      (9   %%3> "  *.A* +66 ) 3 7   ||I&&$h*>O$??GGFh;. !X-"EfjjQXFYEZ [\\!\1--" " "0Ev@HTVW W- H #s7AC7C1 rOrrrr^rtrr{r3) r}rQrKrLrSrMr#rVrer[rs r#rT,AsyncV1FirecrawlApp.async_generate_llms_texts<  A      (9   %%3> "  *.A* +)'"5  '')CQ6;;4;#PQ +G95 ( %11<<. ,   %SV$ $ %s<A-C0#BBBCB B=$B88B==Cc# SSKnURS[SS9 UR5nUR UR SU3U5IShvN $N![ an[[U55eSnAff=f7f)a Check the status of an asynchronous LLMs.txt generation job. .. deprecated:: /v1/llmstxt is deprecated and will not be replaced. Args: id (str): The ID of the generation job Returns: V1GenerateLLMsTextStatusResponse containing: * success (bool): Whether generation completed successfully * status (str): Status of generation (processing/completed/failed) * data (Dict[str, str], optional): Generated text with fields: - llmstxt (str): Generated LLMs.txt content - llmsfulltxt (str, optional): Full version if requested * error (str, optional): Error message if generation failed * expiresAt (str): When the generated data expires Raises: ValueError: If status check fails rNrOrr/r`rr}rr#rers r#rU3AsyncV1FirecrawlApp.check_generate_llms_text_statusRs.  A   '') %00<<. RD1  %SV$ $ %r)rrrKrrm/_AsyncV1FirecrawlApp__experimental_stream_stepsrrr(rrrrrc # SSKn U RS[SS9 0n UbX+S'UbX;S'UbXKS'UbX[S 'UbXkS 'UbX{S '[S0U D6n UR UUUUUUS 9IShvN n U R S 5(aSU ;aU $U Sn SnSnUR U 5IShvN nU(a-SU;a'USUSnUH nU"U5 M [US5nU (a-SU;a'USUSnUH nU "U5 M [US5nUSS:XaU$USS:Xa[SUR S535eUSS:waO [R"S5IShvN M[SSS9$GNNN7fr) r#r=r>rVrrrrrrVr@r_)r}r(rrrKrrmrrrr#rrrBrrrrrrrs r#r!AsyncV1FirecrawlApp.deep_researchxsV  F     *3J '  !+5K (  )1I &  &0?, -  $.;N + & 2O$::6BBF|v5!' !56I6J!K .H)!/&)&*>&?#Y&0$Y/0A0BC )Ff%*$'y(9$:!h;. !X-"? 7@S?T UVV!\1--" " "-0,EAlmmO C* #s8A(F*E=+AF+F,CF,F-FFF)rrrKrrmrc# SSKnURS[SS9 0n UbX)S'UbX9S'UbXIS'UbXYS 'UbXiS 'UbXyS '[S0U D6n UR 5n S U0U R S S S9En S[ 3U S'URURS3U U 5IShvN $N![an [[U 55eSn A ff=f7f)rrNrrr/rrXrQrZrrr(Trrrrr,) r#r=r>rVrrrr^rtrr{r3) r}r(rrrKrrmrr#rrer[rs r#r'AsyncV1FirecrawlApp.async_deep_researchs%D  F     *3J '  !+5K (  )1I &  &0?, -  $.;N + & 2* +4  +4"+4-1cN+4^7:^^"&sCx.^;?S>^^.3^>B38n^27:]]%)#s(^]].3]>B38n],+G,B,B+C+TX+6 Z# Zs Z[^ Zor Zwz Z-(=-,';,/3-1 *9*9]+*9&c] *97L *9^04-1 (@s)(@^,(@&c] (@7L (@\mq04040404&*%*37%)4837(,OS(,.237EI)DMDMd7,g$hij DM d38n- DM #49- DM#49-DM (~DMsmDMc]DM/0DMTNDM$,D>DM#+4.DM ~DM G$JKL!DM" ~#DM$l+%DM&#<0'DM(d57I=ZgivyGIWYrt*$@AB)DM**#.+DMTW[,0,0,0,0"&!&/3!%04/3$(KO*./3AE*.'()--HC3iHC$w(Q RST HC $sCx.) HC tCy) HCtCy)HC$D>HC3-HC#HC+,HCHC (~HC'tnHCD>HC  FGH!HC",'#HC$|,%HC&$u\3E}VceruCESUnp{&{ |}~'HC(')HC* }+HC,"#-HC0 %1HC^W[,0,0,0,0"&!&/3!%04/3$(KO*./3AE*..2)--JI3iJI$w(Q RST JI $sCx.) JI tCy) JItCy)JI$D>JI3-JI#JI+,JIJI (~JI'tnJID>JI  FGH!JI",'#JI$|,%JI&$u\3E}VceruCESUnp{&{ |}~'JI(')JI*&d^+JI,"#-JI0 1JI`.2-1#'-1#/3.2/3)-489=3726,0#+/'()-+uB uB S * uB  S * uB C= uB&c]uB}uB'tnuB&d^uB'tnuB!uB!1uB%_ 456uB#+4.uB "*$!uB"$D>#uB$}%uB&#4.'uB( })uB*"#+uB. /uBx.2-1#'-1#/3.2/3)-489=3726,0#+/'()-+sB sB S * sB  S * sB C= sB&c]sB}sB'tnsB&d^sB'tnsB!sB!1sB%_ 456sB#+4.sB "*$!sB"$D>#sB$}%sB&#4.'sB( })sB*"#+sB. /sBj@3@3H@D)_#)_S#X)__b)_lA)_^!%)--1'+#!&/3(,KE KE KE ! KE %TN KEtnKE}KE#KE+,KE%KE2?KE^)-dR%)$(+/3805+0.2dR49%dRSM dR SM dR $C= dR#+4.dR (~dR#4.dRDcN+dR8I7MdRLB #B :UB H # :O 2 3 3H 2-S-T#s(^-(&%s&%7H7M&%T)-Q%%)$(+/3805+0.2Q%49%Q%SM Q% SM Q% $C= Q%#+4.Q% (~Q%#4.Q%DcN+Q%8I7MQ%n'+-126 DWDWsm DW %TN DW "*$ DW <\ DWT'+-1$(26>%>%sm >% %TN >% D> >%"*$>%%@$%$%@`$%T(,(,&*-1+/:>FJDHgngn } gn ! gn sm gn&c]gn$C=gn*2$gn"(DcN+;T+A"BCgn $sCx.)94)? @AgnNjgnZ(,(,&*-1+/:>C%C% } C% ! C% sm C%&c]C%$C=C%*2$C%DHS>C%J)%3)%;W)%^$(!%$("&%)&*%*8>#ABCN *N N r5rKc ^\rSrSrSrS\S\4U4SjjrSSjrSS jr S \ \\ 4SS4S jr S \ RS \SS4SjrS\S \S\S\S\4 SjrSrU=r$)rxizI Async version of V1CrawlWatcher that properly handles async operations. rrc$>[TU]X5 grrM)r}rrrOs r#r~AsyncV1CrawlWatcher.__init__s !r5ruNc"# [R"URSSURR34/S9IShvN nUR U5IShvN SSS5IShvN gN.NN !,IShvN (df  g=f7f)zK Establishes async WebSocket connection and starts listening for messages. rr)r#Nr$r(s r#r&AsyncV1CrawlWatcher.connects}%% KK!0GDHHBA/BA5A1A5 B)A3*B1A53B5B ;A> < B Bc# UShvN n[R"U5nURU5IShvN M:N5N g7f)z Listens for incoming WebSocket messages and handles them asynchronously. Args: websocket: The WebSocket connection object Nr,r/s r#r'AsyncV1CrawlWatcher._listenr2r3r0c# USS:Xa<SUlURSURURURS.5 g USS:Xa@SUlURSURURUSURS.5 g USS:XamUS S UlURR US R S /55 URH!nURS X RS .5 M# g USS :XaAURR US 5 URS US URS .5 g g 7f)z Handles incoming WebSocket messages based on their type asynchronously. Args: msg (Dict[str, Any]): The message to handle rrrfr?rrgr@rArrrrBNrCrDs r#r.#AsyncV1CrawlWatcher._handle_messagerGrHrrcf# UR5IShvN nURSS5nURSS5nUR R URX$U5IShvN n[R"U5eNm! [R"SUR35e=fND7f)z) Handle errors from async API responses. Nrr~rrrf)rHrrSrWrrrgrhs r#r!AsyncV1CrawlWatcher._handle_errors {'}}.J&NN74PQM&NN96]^M 99(//6bopp!!'**/ {%%(ghphwhwgx&yz zqs8B1BB(B*B1*B/+B1B%B,,B1rrrc.# URXX45$7frlrmrs r#rg,AsyncV1CrawlWatcher._get_async_error_messagerorp)rrI)r-r.r/r0r1r3rKr~r&r'rrr.rSrrrrgr4rrs@r#rxrxs"3"%8"* ,Rc3hRDR,+G,B,B+C+TX+ Z# Zs Z[^ Zor Zwz Z Zr5rx)Tr1loggingrr?typingrrrrrrr r r rHr rrrLr%rSrV getLoggerrLoggerr2r$rr% BaseModelr'r7r=rBrOr^rcrlrsrrrrrrrrrrrrrrrrrr r rrrrr r'r1r3r9r;r?rArGrJrMrOrVr]r_rdrfrjrrrrKrxr,r5r#rs[  XXX  !++K88  - CLJ!X''! (H..(h(( C8--C:(,,gaj: *x))* Uh((Uh00$h(($(#8%%# "++" H&& H&&  H&& #X''# X'' 2 2 "($$"(X''(+8%%+._. *1-wqz ,H.., $("4"4 $+H&&+( h((  $H.. $8%%H.. 0($$ 0 H&& 4h(( 4 &**GAJ & 4X'' 4 x)) -**- H..-x))- 8--H$6$6 1h&8&818#5#5 0X%7%701x111 68-- 6&X//&&8#5#5&"&!3!3&&8+=+=&&x'9'9& x))  +h(( +X+ X+ vVYRYRvx .x t;OZ.OZr5