# Live Site Validation — Progressive Escalation Ladder

For web scraping / API integration spikes: test every target site with escalating approaches until you find what works. Do NOT assume a site is scrapable because `web_extract` returned data — web_extract uses residential proxies your server doesn't have.

## The ladder (test in order — stop when one works)

```
1. aiohttp+BS4 (plain HTTP, real Chrome headers)
   → Check: status 200, body >5KB, listing indicators visible
   → Kill: 403/429, captcha in body[:500], body <2KB (JS shell)

2. aiohttp + stealth headers (Accept, Sec-Fetch-*, Sec-Ch-Ua-*)
   → Adds full browser-like HTTP headers

3. aiohttp + Googlebot UA
   → Some sites allow-All for search crawlers
   → Kill: PerimeterX — checks more than UA

4. Playwright headless
   → Signal: rendered HTML with listing data
   → Kill: PerimeterX, Akamai, DataDome detect headless

5. Playwright headless + stealth init_script
   → navigator.webdriver removal, canvas/WebGL spoofing, chrome.runtime patch
   → Kill: advanced behavioral fingerprinting

6. Playwright non-headless via xvfb-run
   → Avoids headless detection
   → Kill: IP reputation/ASN blocks (datacenter IPs)

7. Patchright (Playwright fork for PerimeterX — patches CDP leaks)
   → Try headless AND non-headless
   → Kill: PerimeterX still detects automated browser on datacenter IPs

8. FlareSolverr proxy
   → Works on: Cloudflare challenges
   → Does NOT work on: PerimeterX, Akamai, DataDome

9. Direct API endpoint probing
   → Test: /search, /api/, /graphql, /ajax/, /v1/, /v2/, /_next/data/
   → Also check robots.txt for sitemap URLs, disallowed API paths
   → Kill: all endpoints also 403

10. Sitemap → individual pages
    → Get listing URLs from sitemap XML
    → Then test each URL with layers 1-3
    → Kill: individual pages also blocked (Zillow)

11. Alternative platform discovery
    → Search for weaker competitors (less anti-bot, JS-shell-only, scrapable byproduct)
```

## Quick probe script

```python
import aiohttp, asyncio

async def probe(url, name):
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36'}
    try:
        async with aiohttp.ClientSession(headers=headers) as s:
            async with s.get(url, timeout=aiohttp.ClientTimeout(total=10)) as r:
                body = await r.text()
                blocked = any(x in body[:500].lower() for x in [
                    'captcha', 'access denied', 'perimeterx', 'cf-browser-verification', 'px-captcha'
                ])
                js_shell = len(body) < 5000
                verdict = 'BLOCKED' if blocked else 'JS-SHELL' if js_shell else 'OK'
                print(f'{r.status} | {name:<20} | {verdict:>8} | {len(body):>8} B')
    except Exception as e:
        print(f'ERROR | {name:<20} | {str(e)[:50]}')

asyncio.run(probe('https://www.zillow.com/homes/for_rent/Boston-MA/', 'Zillow'))
asyncio.run(probe('https://boston.craigslist.org/search/apa', 'Craigslist'))
asyncio.run(probe('https://www.redfin.com/city/1826/MA/Boston/apartments-for-rent', 'Redfin'))
asyncio.run(probe('https://www.apartments.com/boston-ma/', 'Apartments.com'))
```

## Block-type signatures

| Block type | HTTP status | Body signature | AppID/Key | What detects you |
|------------|-------------|----------------|-----------|-----------------|
| **PerimeterX** | 403 | `px-captcha`, `PXHYx10rg3`, `Access to this page has been denied`, `_pxVid` | `PXHYx10rg3` | Browser fingerprint + behavior + IP |
| **Akamai** | 403 | `Access Denied`, `Reference #` | — | IP + header fingerprint |
| **Cloudflare** | 403 | `cf-browser-verification`, `jschl-answer`, `Checking your browser` | — | TLS fingerprint + IP |
| **DataDome** | 403 | `dd`, `datadome` | — | JS fingerprint + IP |

## 2026-07-06 Live Spike Results: US Rental Platforms

9 platforms tested through 15 bypass attempts across the full escalation ladder:

| Platform | Layer 1 HTTP | Layer 5 xvfb | Layer 7 Patchright | Layer 8 FS | Layer 9 API | Layer 10 Sitemap | Verdict |
|----------|-------------|--------------|-------------------|------------|-------------|-----------------|---------|
| **Craigslist** | ✅ 200, 328 results, full data | — | — | — | — | — | **BUILD** |
| **Redfin** | ✅ 200, 3.3MB HTML, 32+ listings, prices/BR, amenities, phone | — | — | — | ✅ reactServerState | — | **BUILD** |
| **Zillow** | ❌ PX | ❌ PX | ❌ PX | ❌ PX | ❌ 403 | ✅ URLs found, pages blocked | Needs residential proxy |
| **Apartments.com** | ❌ Access Denied | — | — | ❌ Denied | ❌ 403 | — | Needs residential proxy |
| **HotPads** | ❌ PX (Zillow-owned) | — | — | — | — | — | Same as Zillow |
| **Trulia** | ❌ PX (Zillow-owned) | — | — | — | — | — | Same as Zillow |
| **Realtor.com** | ⚠️ 429 rate-limited | ⚠️ 1.8KB JS shell | — | — | — | — | Rate-limited |
| **Rent.com** | ⚠️ 200 JS shell | ⚠️ 860KB, 0 prices | — | — | ❌ GraphQL bot detect | — | Dead |
| **Zumper** | ⚠️ 3KB shell | ❌ Timeout 30s | — | — | — | — | Dead |
| **Facebook Marketplace** | ❌ 400 needs login | — | — | — | — | — | Needs auth |

## Key lessons

1. **web_extract ≠ local HTTP.** Hermes' web_extract tool uses Firecrawl's residential proxy infrastructure. It can access Zillow (15K+ listings). Your server's aiohttp cannot. Never declare a site scrapable based on web_extract results.

2. **PerimeterX cannot be beaten headless.** 15 attempts (aiohttp, Playwright×5 variants, FlareSolverr, Patchright×2, Googlebot, undetected-chromedriver, sitemaps, Next.js SSR, GraphQL, mobile API, internal API) all failed. Every GitHub repo claiming to bypass PX either uses paid residential proxies or requires manual human captcha solving.

3. **Plan AFTER spiking, not before.** The original apt-finder plan included Zillow and Apartments.com scrapers before live validation. This took 6 spike rounds to correct. Spike first, plan the approach that actually works.

4. **Build what works, skip what doesn't.** Craigslist (328 results) + Redfin (32+ results, richer data) is a solid two-platform aggregator. Adding dead-on-arrival scrapers makes the tool fail silently.