---
name: streaming-source-aggregator
category: software-development
tags: [streaming, scraper, flask, torrent, p2p, kodi, embed, video]
description: Build a unified streaming backend that aggregates multiple free sources (torrents, embeds, direct video) into one API + frontend. Covers Flask backend architecture, TorrServer P2P integration, free scraper patterns from Kodi addons (Scrubs V2, Fen, Umbrella), and ad-free playback strategies.
triggers:
  - "build a streaming web app"
  - "combine all scrapers"
  - "free movie streaming backend"
  - "torrent streaming server"
  - "kodi addon reverse engineering"
  - "unified streaming sources"
---

# Streaming Source Aggregator

Build a unified backend + frontend that aggregates ALL free streaming sources into one system.

## Architecture

```
Flask Backend (:8800)
  ├─ /api/search          TMDB movie search
  ├─ /api/movie/<id>      Movie details (cast, genres, ratings)
  ├─ /api/torrents        Torrent sources (TPB, BitSearch)
  ├─ /api/free_sources    ALL sources combined (torrents + direct + embeds)
  ├─ /api/add_magnet      Add magnet to TorrServer (separate from stream!)
  ├─ /stream/<hash>       Proxy TorrServer stream to browser
  ├─ /proxy?url=...       Proxy direct video URLs (avoids CORS/ad injection)
  ├─ /api/ts_status       TorrServer health
  ├─ /api/torrent_status  Torrent download status
  └─ /*                   Serves React/Vite frontend (e.g. SanuFlix) or static HTML

Frontend (React/Vite or static HTML)
  ├─ Movie search + grid with posters
  ├─ Movie detail page (synopsis, cast, rating)
  └─ Watch page with unified source grid + video player
```

## Free Sources (from Kodi addon RE)

### Tier 1 — Reliable
| Source | Type | How | Notes |
|--------|------|-----|-------|
| **TPB** (apibay.org) | Torrent | JSON API | Returns info hashes + seeders. Most reliable free source. |
| **BitSearch** | Torrent | HTML scrape | Secondary torrent source. |
| **2Embed** | Embed | iframe URL | Reliable embed, minimal ads. |
| **AutoEmbed** | Embed | iframe URL | Reliable embed, clean. |

### Tier 2 — Unreliable / Dead
| Source | Type | Why unreliable |
|--------|------|----------------|
| GDrivePlayer | Direct | API changes frequently |
| Ronemo | Direct | Site often down |
| DoodStream | Direct | Blocks server scraping, Cloudflare |
| VidCloud | Direct | HTML structure changes |
| StreamHide | Direct | Blocks scraping |
| Bing Search | Discovery | Random results, low hit rate |

### Tier 3 — Additional (from Kodi addons)
- VidSrc, EmbedSu — embeds with ads
- StreamWish, VidLink, Voxzer, FurHer — free hosts, often dead
- Gomo, Ronemo, LinkBin — similar free hosts

## TorrServer Integration

### Critical: Chicken-and-Egg Problem
TorrServer only starts downloading a torrent when it has an ACTIVE reader on the `/play/{hash}/{file_id}` endpoint. The frontend MUST set the stream URL immediately and let the browser connect, rather than polling for `files > 0` before setting the URL.

**WRONG** (polls, never starts):
```python
# Frontend polls for files > 0 - TorrServer never starts because no reader
# WRONG approach - chicken-and-egg
```

**RIGHT** (sets URL immediately):
```python
# Frontend sets stream URL right away
# Flask proxy:
# 1. Adds magnet to TorrServer
# 2. Immediately calls /play/{hash}/{fid}
# 3. Streams whatever comes back
# TorrServer sees reader, starts downloading
```

### Stream Endpoint Flow
1. Call `/api/add_magnet?hash=XXX` — adds magnet to DB (does NOT consume stream data)
2. Browser requests `/stream/XXX` — Flask calls TorrServer's `/play/XXX/1`
3. TorrServer connects to peers, downloads metadata, streams data
4. Flask proxies data to browser as `Content-Type: video/mp4`, `Content-Disposition: inline`

### Auto-Detecting Video File
When `file_stats` are available:
- Look for files ending in `.mp4`, `.mkv`, `.avi`, `.mov`, `.webm`, `.m4v`
- Pick the LARGEST video file (by `length` field)
- If no video match, use first file as fallback

If content-type is `text/*`, try file_id 2, then 1.

### Separate Magnet Add from Stream
**CRITICAL**: The magnet add and stream endpoints MUST be separate.
- `/api/add_magnet?hash=XXX` — just adds the magnet, returns JSON
- `/stream/XXX` — only called by the browser's `<video>` tag

If you `fetch(/stream/XXX)` from JavaScript to trigger the magnet add, the Python code consumes the stream data before the browser can use it.

### Torrent Status States
From TorrServer MatriX fork:
- `"Torrent in db"` — saved to DB, not active. Need a stream reader to activate.
- `"Torrent getting info"` — connecting to DHT/trackers
- `"Torrent working"` — downloading, streaming possible
- `stat: 0` = stored; `stat: 1` = getting info; `stat: 2` = working

## Embed Source Ads Problem

**You CANNOT block ads in embed iframes server-side.** The embed sites serve ads as part of their page. Sandbox attribute (`sandbox="allow-scripts"`) breaks the embed functionality.

**Why Android apps don't have ads:** They load the embed page in a hidden WebView, intercept network requests to extract the direct `.mp4`/`.m3u8` video URL, then play in a native `ExoPlayer` / `VideoView` — bypassing the ad HTML entirely. This requires a full headless browser server-side.

**Best options for ad-free:**
1. P2P torrents via TorrServer (self-hosted, zero ads)
2. A browser with ad blocking on the client device
3. Paid debrid service (Real-Debrid, Premiumize)

## Frontend Source Display

Show ALL sources in one unified grid (not separate sections):
- **Torrent** sources: `P2P 720s 1080p` (seeders + quality)
- **Direct** sources: `GDrive 1080p` (source + quality)
- **Embed** sources: `2Embed` (source name only)

Color-code by type: green for torrent/P2P, indigo for direct, gray for embed.
Show "Show all N sources" button if >6 sources.

## Free Scraper v2 Architecture

When direct HTTP scraping is needed (extracting m3u8/MP4 URLs from embed sites), use a layered extractor approach with fallbacks:

### Extractor Priority Chain
```
1. AJAX API extractor (highest priority) — calls site's internal API directly
   - VidSrc: /ajax/embed/episode/{dataId}/sources → /ajax/embed/source/{sourceId}
   - RC4 decryption key: 'WXrUARXb1aDLaZjI' (expires, check vidsrc-new repo for updates)
   
2. Playwright headless browser — loads page as real browser, intercepts network requests
   - Catches .m3u8, .mp4, /master URLs from ALL network traffic
   - Requires puppeteer-extra + puppeteer-extra-plugin-stealth for detection bypass
   - Still blocked by sites that check window.top === window (iframe check)
   - ~114MB Chromium download, ~500MB RAM at runtime
   
3. Generic HTTP (lowest priority) — regex search in HTML, inline scripts, iframe follows
   - Patterns: m3u8, mp4, master, index.m3u8
   - Follow iframe src, search recursively
```

### Why Server-Side Extraction Fails
Most embed sites use this anti-bot pattern that no server-side scraper can fully bypass:
```
1. if(window===window.top) redirect    → blocks direct access, requires iframe
2. navigator.webdriver check           → detects headless automation
3. JavaScript-rendered video URL       → never in static HTML
4. Click-to-play gate                  → video URL requires user interaction
5. Cloudflare challenge                → requires real browser JS execution
```

### The Only Reliable Approaches
1. **Debrid services** ($3/mo) — instant cached torrents, zero anti-bot
2. **Browser-based** (Playwright + stealth) — heavy, ~50% success rate on modern sites
3. **P2P torrents via TorrServer** — free, works, takes 15-30s peer connection
4. **Android native WebView intercept** — the approach Cinema HD, BeeTV, TeaTV use (see `references/android-app-scraping-architecture.md`)

## Merged Scraper Stack

The complete system has three services that work together:

```
Flask Backend (:8800)              — Frontend + TMDB + TPB + embeds + TorrServer proxy
ultimate-scraper (:3091)            — Ultimate scraper v4.1 (10 sources + multi-debrid + Stremio protocol)
free-scraper (:3092)                — Playwright-based embed URL extraction
```

The `ultimate-scraper` (Node.js Express app at `~/ultimate-scraper/`) has a custom `ultimate.js` scraper module that runs ALL scrapers in parallel, deduplicates by hash, and optionally checks RD cache.

### Ultimate Scraper Module (`src/scrapers/ultimate.js`) — v4.1

10 scrapers running in parallel:
- **TPB** (apibay API) — always works, most reliable
- **Nyaa.si** (anime — cheerio table parse, real titles/seeders/sizes)
- **AniDex** (anime — JSON API)
- **SubsPlease** (anime — JSON API)
- **AnimeTosho** (anime — RSS HTML)
- **YTS** (JSON API) — often blocked server-side
- **1337x** (proxy API + direct HTML fallback)
- **BitSearch** — direct HTML scrape
- **TokyoTosho** (anime — cheerio HTML parse)
- **Torznab** (Jackett/Prowlarr — if JACKETT_URL + JACKETT_API_KEY configured)

Wired in-memory cache (10min TTL) via `getCached`/`setCached`. Results deduplicated by infoHash.

### Multi-Debrid Integration (4 providers)

instantAvailability is **DEPRECATED** (error 37, mid-2024). The correct flow for all providers is addMagnet → poll → unrestrict.

| Provider | Resolve Endpoint | API Pattern |
|----------|-----------------|-------------|
| Real-Debrid | `/api/rd/resolve?hash=X` | addMagnet → selectFiles → poll info → unrestrict/link |
| AllDebrid | `/api/ad/resolve?hash=X` | magnet/upload → status poll → link/unlock |
| Premiumize | `/api/pm/resolve?hash=X` | directdl check (instant if cached) → transfer/create → poll → directdl |
| TorBox | `/api/tb/resolve?hash=X` | createtorrent → mylist poll → requestdl |
| **Auto** | `/api/debrid/resolve?hash=X` | Tries all enabled providers in order |

Config auto-detection: set `RD_API_KEY`, `AD_API_KEY`, `PM_API_KEY`, `TB_API_KEY` in `.env` to enable each.

### TorrServer Chicken-and-Egg Fix

**CRITICAL**: TorrServer only starts downloading a torrent when it has an ACTIVE reader on the `/play/{hash}/{file_id}` endpoint. The frontend MUST set the stream URL immediately rather than polling for `files > 0`.

**WRONG** (polls, never starts):
```javascript
// Poll for files > 0 — TorrServer never starts because no reader connected
// files stays at 0 forever → infinite loop
```

**RIGHT** (sets URL immediately):
```javascript
// 1. Call /api/add_magnet?hash=XXX to add magnet
// 2. Set stream URL right away: <video src="/stream/XXX">
// 3. Flask calls TorrServer's /play/XXX/1
// 4. TorrServer sees reader, starts downloading, serves data
// 5. Browser plays as data arrives
```

These apps DO NOT use Stremio addons (Torrentio, Comet). They ARE the scraper — everything is compiled into the APK.

### Flow
```
         ┌──────────────┐
         │  Torrent     │  1337x, TPB, YTS, EZTV,
         │  Site        │  TorrentGalaxy, Kickass,
         │  Scrapers    │  LimeTorrents (15-20 sites)
         └──────┬───────┘
                │ info hashes
                ▼
         ┌──────────────┐
         │  Real-Debrid │  POST /torrents/instantAvailability
         │  API Check   │  (or: addMagnet → selectFiles → poll → unrestrict)
         └──────┬───────┘
                │ HTTP link from RD CDN
                ▼
         ┌──────────────┐
         │  ExoPlayer   │  Direct playback from RD's servers
         └──────────────┘
```

### Why They Don't Need Stremio Addons
- **Built-in scrapers** — 15-20 torrent site scrapers compiled into the APK itself (Kotlin/Java)
- **Direct hash extraction** — scrape pages from 1337x, YTS, EZTV, TPB, etc., extract info hashes
- **RD API integration** — call `instantAvailability` (deprecated) or fallback to `addMagnet` → poll → `unrestrict`
- **Native player** — play RD's direct HTTP link in ExoPlayer/Media3, zero ads

### Non-Debrid Path
For users without debrid:
- Direct HTTP scrapers for free hosts (StreamTape, DoodStream, MixDrop, FileMoon)
- Same extractor pattern as our free-scraper, but running in **WebView** on device
- WebView executes JavaScript, bypasses Cloudflare naturally
- App intercepts network requests to capture direct video URLs
- Falls back to browser-based extraction (the user's real browser handles the anti-bot)

### Key Difference from Server-Side Scraping
| Approach | Runs where | Bypasses Cloudflare? | Success Rate |
|----------|-----------|---------------------|-------------|
| Server-side HTTP | Server | ❌ Blocked | ~10% |
| Playwright headless | Server | 🟡 Sometimes | ~50% |
| **WebView on device** | **User's phone** | **✅ Yes** | **~95%** |
| Native ExoPlayer + RD | Phone | ✅ Yes | 100% |

The WebView approach is the secret sauce — it runs on the user's actual device with a real browser engine,
so Cloudflare can't distinguish it from a normal user. The app loads the embed page in a hidden WebView,
intercepts ALL network requests, and captures the `.m3u8`/`.mp4` URLs. Then it stops the WebView and
plays the URL in the native player. This is impossible to replicate server-side without running a full
browser with a real user profile.

### Relation to Our System
- Our free-scraper's **Playwright extractor** is the server-side equivalent of the WebView approach
- The difference: Playwright runs on the server (IP reputation matters), WebView runs on the user's device
- For a truly reliable free solution, the extraction MUST happen on the client device, not the server

### See Also
- `references/android-app-scraping-architecture.md` — Detailed flow diagrams, RD API call sequence, scraper patterns extracted from decompiled APKs

## SanuFlix React Frontend Integration

Replace Flask Jinja2 templates with a React/Vite SPA for a better UI:

### Setup
```bash
git clone <sanuflix-repo>
cd sanuflix && npm install
# Set env vars in .env:
#   VITE_TMDB_API_KEY=<key>
npm run build  # Outputs to dist/
```

### Flask serves static build
```python
DIST = '/path/to/sanuflix/dist'
@app.route('/')
@app.route('/<path:path>')
def serve(path='index.html'):
    fp = os.path.join(DIST, path)
    if path and os.path.isfile(fp):
        return send_from_directory(DIST, path)
    return send_from_directory(DIST, 'index.html')
```

### Key Frontend Modifications
1. **Remove snowflake effects** — delete ChristmasSnow import, snow CSS animations
2. **Unified source grid** — show ALL source types (torrent + embed + direct) in one place
3. **Player**: `<video>` tag for direct URLs (mp4/m3u8), iframe for embed URLs
4. **P2P flow**: Show loading overlay for 8s, then set stream URL (don't poll for files)
5. **Source labels**: `P2P 720s 1080p` for torrents, `2Embed` for embeds, `GDrive` for direct
6. **"Show all N sources"** button when >6 sources

### Updating the Server Config
```typescript:src/config/servers.ts
// Only sources that actually work
export const STREAMING_SERVERS = [
  { id: '2embed', name: '2Embed', movieUrlTemplate: 'https://www.2embed.cc/embed/{tmdbId}' },
  { id: 'autoembed', name: 'AutoEmbed', ... },
  { id: 'vidsrc', name: 'VidSrc', ... },
  { id: 'embedsu', name: 'EmbedSu', ... },
]
```

### Pitfalls

#### P2P Streaming NOT Working
1. **Wrong file_id** — file 1 is often a subtitle/text file. Auto-detect video files by extension.
   - Check `file_stats` for video extensions (`.mp4`, `.mkv`, `.avi`, `.mov`, `.webm`, `.m4v`)
   - Pick the LARGEST video file
   - If content-type from TorrServer is `text/*`, try file_id 2 then 1
2. **Browser downloads instead of plays** — missing `Content-Disposition: inline` header. Flask must add this to the stream response.
3. **Torrent never starts (chicken-and-egg)** — TorrServer only downloads when it has an ACTIVE reader on `/play/{hash}/{file_id}`. The frontend MUST set the stream URL immediately (don't poll for `files > 0`).
4. **TPB hashes are old/dead** — many TPB results have no real seeders. Filter by `seeds > 0` but even then some are stale.
5. **TorrServer needs open ports** — on some networks, can't connect to DHT/trackers. Symptom: torrents stuck on "Torrent getting info" with 0 peers.
6. **Separate magnet add from stream** — NEVER `fetch(/stream/<hash>)` from JavaScript to trigger a magnet add. The Python proxy consumes the stream data before the browser can use it. Use a dedicated `/api/add_magnet?hash=XXX` endpoint instead.

### Embed Sources
1. **Sandbox breaks everything** — don't use sandbox for embed iframes. Mark problematic sources in the UI instead.
2. **Free embed hosts change URLs** — maintain a list of working domains and test them periodically.
3. **Cloudflare blocks scraping** — use browser User-Agent headers and handle 403 errors gracefully.

### General
1. **API rate limiting** — TMDB rate-limits at ~50 req/s. TPB (apibay.org) has no rate limit but may go down.
2. **SSL certificate issues** — use `ssl._create_unverified_context` for scraping, but prefer verified where possible.
3. **Unicode in torrent names** — always use `utf-8`, `errors='replace'` when decoding.
4. **React state timing** — `setActiveUrl` in React fires async. Use `useEffect` with dependency, not `setTimeout`.

### See Also
- `references/free-source-providers.md` — Complete list of free video host domains from Scrubs V2 and other Kodi addons
- `references/torrserver-api.md` — TorrServer REST API reference (MatriX fork endpoints, streaming URLs, status codes)
- `references/scraper-techniques-research.md` — Research findings from 10+ scraper projects: Playwright headless, AJAX API extraction, RC4 decryption, anti-bot bypass techniques, and the reliability reality
- `references/android-app-scraping-architecture.md` — How Cinema HD / BeeTV / TeaTV scrape without Stremio addons
- `references/merged-scraper-architecture.md` — How Flask, nexstream-scraper, and free-scraper coexist + adding new scrapers + RD API flow + Torrentio provider list