# Scraper Techniques Research

Research conducted July 2026 — analysis of 10+ scraper projects to extract best practices.

## Repos Analyzed

| Repo | Technique | Language | Key Insight |
|------|-----------|----------|-------------|
| DivineChile/vidsrc-scraper | Playwright headless | Node.js | Intercept ALL network requests via `page.route('**/*')`, capture `.m3u8` from live traffic. Click video frame to trigger load. |
| scara78/vidsrc-new | AJAX API + RC4 decrypt | Node.js | Call site's internal API directly: `/ajax/embed/episode/{id}/sources` → `/ajax/embed/source/{id}` → RC4 decrypt with key `'WXrUARXb1aDLaZjI'` |
| codyklr/M3U8-Scraper | Selenium + FFMPEG | Python | Uses full browser to bypass Cloudflare, then extracts m3u8 from devtools network log |
| reemuree/iptv-m3u8-auto-scraper | Regex brute force | Python | Scans HTML + JS + JSON + AJAX for m3u8 patterns. Follow iframes recursively. |
| pratikkarbhal/m3u8_StreamSniper | Chrome DevTools Protocol | Python | Uses CDP to capture m3u8 from Chrome's network tab via GitHub Actions |
| Scrubs V2 (jewbmx/repo) | Search engine + direct host scraping | Kodi/Python | Uses Bing/DuckDuckGo/Google to discover video links. 50+ video host domains with fallback logic. |
| Fen Light+ | Debrid + free scrapers | Kodi/Python | Parallel scraper execution, caching layer, quality detection |
| Seren | Provider packages | Kodi/Python | Provider-based architecture with installable scraper packages |
| Jacktook | Jackett/Prowlarr + TorrServer | Kodi/Python | External indexer aggregation via Torznab protocol |

## Extractor Priority Chain (most → least effective)

1. **AJAX API** — Call site's internal API directly. Fastest, no browser needed. Breaks when API changes.
2. **Playwright headless + stealth** — Real browser execution. Handles Cloudflare, JS rendering. ~50% success rate on modern anti-bot sites.
3. **Generic HTTP** — Regex search in HTML, inline scripts, iframe recursion. Works on simple sites, fails on JS-heavy ones.
4. **Selenium/Chrome DevTools Protocol** — Heavier than Playwright, same detection issues.
5. **Puppeteer-extra + stealth** — Best anti-detection plugin available. Still blocked by `window.top === window` checks.

## Anti-Bot Detection Techniques Used by Embed Sites

| Technique | How to Bypass |
|-----------|---------------|
| `if(window===window.top) redirect` | Impossible server-side. Requires real browser in iframe context. |
| `navigator.webdriver` check | `puppeteer-extra-plugin-stealth` deletes this property |
| Chrome DevTools Protocol detection | Stealth plugin patches CDP flags |
| User-Agent fingerprinting | Use latest Chrome UA string |
| `navigator.plugins` length check | Stealth plugin populates with realistic plugins |
| WebGL/Vendor fingerprinting | Harder to spoof, changes by GPU |
| Cloudflare challenge | Requires real browser JS execution |
| Click-to-play gate | Must interact with page (click play button) |
| Referer header check | Set proper Referer header matching the embed context |

## Working Free Sources (as of July 2026)

### Reliable
- **The Pirate Bay** via `https://apibay.org/q.php?q={title}&cat=0` — Returns JSON with info_hashes, seeders, leechers. No auth needed.
- **BitSearch** via `https://bitsearch.to/search?q={title}&category=1` — HTML scrape for info hashes.
- **2Embed** — `https://www.2embed.cc/embed/{tmdb_id}` — iframe embed, works reliably.
- **AutoEmbed** — `https://autoembed.cc/embed/movie/{tmdb_id}` — iframe embed, works reliably.

### Unreliable/Dead
- GDrivePlayer API — API frequently changes endpoints
- Ronemo — Site often down
- DoodStream — Blocks server-side scraping with Cloudflare
- VidCloud — HTML structure changes often
- StreamHide — Actively blocks scraping
- Peerflix, PirateBay+ (Stremio addons) — Dead
- Torrentio, Comet, MediaFusion — All Stremio addons confirmed dead

## The Reality

No server-side scraping technique can fully overcome modern anti-bot protection. The only approaches that consistently work are:
1. **Paid debrid** ($3/mo) — instant cached torrents
2. **P2P via local TorrServer** — free, ad-free, takes 15-30s
3. **Client-side ad blocker** — uBlock Origin on the user's device