# Rental / Apartment Site Scraping Patterns

Research verified July 6, 2026 via **live spikes against production sites** (15 bypass attempts across 9 platforms). Site behavior changes — re-validate with `curl` before building.

## VERIFIED WORKING — Craigslist

| Property | Value |
|---|---|
| **Browsing method** | aiohttp + BeautifulSoup4 ✅ |
| **Live test result** | 200 OK, 328 `.cl-static-search-result` items, full listing data |
| **Search URL pattern** | `https://{city}.craigslist.org/search/apa?max_price={n}&min_bedrooms={n}&sort=date&availabilityMode=0` |
| **JS required** | No |
| **Auth required** | No |
| **Rate limits** | Moderate. Add 1-2s delays between pages. |
| **Data per listing** | Title, price, address, bedrooms, URL |
| **Selector** | `.cl-static-search-result` |

## VERIFIED WORKING — Redfin

| Property | Value |
|---|---|
| **Browsing method** | aiohttp + BeautifulSoup4 ✅ |
| **Live test result** | 200 OK, 3.3MB HTML, 32+ apartment links, prices per BR, amenities, phone numbers |
| **Search URL pattern** | `https://www.redfin.com/city/{id}/{State}/{City}/apartments-for-rent` |
| **JS required** | No — data in server-rendered HTML cards |
| **Data per listing** | Price per bedroom (Studio: $X, 1bd: $Y), address, amenities (pets, parking), phone, URL |
| **Card text extraction** | `soup.find_all('a', href=re.compile(r'/apartment/'))` then parse parent card text with pipe-separator for structured fields |

## HARD-BLOCKED — Zillow (PerimeterX)

| Property | Value |
|---|---|
| **All bypass attempts failed (15 total)** | ❌ |
| **PerimeterX App ID** | `PXHYx10rg3` |
| **What was tried** | aiohttp → 403, Playwright headless → 403, Playwright stealth → 403, Playwright non-headless xvfb → 403, Playwright mobile UA → 403, Googlebot UA → 403, FlareSolverr → PX unsolved (FS only solves Cloudflare), undetected-chromedriver → hangs, **Patchright** (Playwright fork for PX) headless → 403, Patchright non-headless → 403, Sitemap detail pages → 403, Next.js `_next/data` SSR routes → blocked, GraphQL endpoint → 403, Mobile API endpoints → 403, Internal search API → 403 |
| **Sitemap access** | ✅ Sitemap index returns listing URLs (`/xml/sitemaps/us/hdp/for-rent/sitemap-XXXX.xml.gz`). But every individual listing page is also 403. |
| **Only working approach** | Paid residential proxy (Firecrawl, ScrapingBee, ZenRows — $50-200+/mo) OR web_extract tool via Hermes (uses Firecrawl proxy under the hood) |
| **DO NOT** | Write a Zillow scraper using Playwright. It will not work on a residential IP. Do not include Zillow in standalone CLI plans. |

## HARD-BLOCKED — Apartments.com (Akamai)

| Property | Value |
|---|---|
| **All bypass attempts failed** | ❌ |
| **What was tried** | aiohttp → 403 Access Denied, FlareSolverr → denied, API endpoints (`/search/v2`, `/ajax/search`) → all 403 |
| **Only working approach** | Paid residential proxy or paid scraping API |
| **DO NOT** | Include Apartments.com in any scraper that runs from a residential/datacenter IP. |

## HARD-BLOCKED — Other platforms

| Platform | Status | Reason |
|---|---|---|
| **HotPads** | ❌ 403 | Zillow-owned, same PerimeterX |
| **Trulia** | ❌ 403 | Zillow-owned, same PerimeterX |
| **Realtor.com** | ❌ 429 | Rate-limited, 1.8KB JS shell |
| **Rent.com** | ⚠️ 200 but useless | JS shell, GraphQL API exists (`/graphql`) but returns bot detection |
| **Zumper** | ❌ Timeout | JS shell, 30s timeout on data load |
| **Facebook Marketplace** | ❌ 400 | Requires logged-in session |

## What to actually build

Only Craigslist + Redfin work via plain HTTP. Any multi-platform scraper MUST spike-validate each target BEFORE writing a single line of scraper code. The extractor registry pattern (from free-scraper) allows adding/removing platforms without touching the core aggregator.

| Layer | Platform | Method |
|---|---|---|
| CLI scraper | Craigslist | aiohttp+BS4 ✅ |
| CLI scraper | Redfin | aiohttp+BS4 ✅ |
| Hermes skill | Zillow (bonus) | `web_extract` tool (Firecrawl residential proxy) |

## Source Reliability Scoring (for aggregator ranking)

| Factor | Weight | Notes |
|---|---|---|
| Freshness (hours since listed) | 30% | Craigslist can be minutes-old; Zillow may be hours/days stale |
| Price match to user budget | 25% | Exact match > under budget > over budget |
| Location match | 20% | Within search radius |
| Amenity match | 10% | Pets, parking, laundry, gym — per user preference |
| Platform reliability | 10% | Scraped data quality: Craigslist raw > Redfin clean > Zillow clean > Apartments.com (needs browser) |
| Photo count | 5% | More photos = more established listing, less likely to be scam |

## Existing Repos in This Space (July 2026)

| Repo | Stars | US Sites | Verdict |
|---|---|---|---|
| VikParuchuri/apartment-finder | 1.1k | Craigslist only | Archived 2016. Slack bot pattern still useful. |
| flathunters/flathunter | 1k | None (German) | Mature codebase, Telegram notifier, Selenium. Adapt site scrapers for US. |
| mory91/rent-finder | 2 | Craigslist only | Overengineered LLM agent wrapper. Unfinished. Borrowed model schemas. |
| mmynk/apartments-scraper | 0 | Apartments.com | Selenium-based. Scoring/ranking logic reusable. |
| adinutzyc21/apartments-scraper | 0 | Apartments.com | BS4-only. Shows HTML approach can partially work. |
| hanzili/hanzi-browse apt-finder | 167 | Zillow/Apt/CL/Redfin | Prompt/skill for browser agent, not code. Workflow design (Search→Compare→Contact) is excellent. |
| Janchaloupka/web-scraper-nabidek | 100 | None (Czech) | Clean config-based Discord notifier pattern. |