---
name: streaming-source-aggregator
category: software-development
tags: [streaming, scraper, flask, torrent, p2p, kodi, embed, video]
description: Build a unified streaming backend that aggregates multiple free sources (torrents, embeds, direct video) into one API + frontend. Covers Flask backend architecture, TorrServer P2P integration, free scraper patterns from Kodi addons (Scrubs V2, Fen, Umbrella), and ad-free playback strategies.
triggers:
  - "build a streaming web app"
  - "combine all scrapers"
  - "free movie streaming backend"
  - "torrent streaming server"
  - "kodi addon reverse engineering"
  - "unified streaming sources"
---

# Streaming Source Aggregator

Build a unified backend + frontend that aggregates ALL free streaming sources into one system.

## Architecture

```
Flask Backend (:8800)
  ├─ /api/search          TMDB movie search
  ├─ /api/movie/<id>      Movie details (cast, genres, ratings)
  ├─ /api/torrents        Torrent sources (TPB, BitSearch)
  ├─ /api/free_sources    ALL sources combined (torrents + direct + embeds)
  ├─ /api/add_magnet      Add magnet to TorrServer (separate from stream!)
  ├─ /stream/<hash>       Proxy TorrServer stream to browser
  ├─ /proxy?url=...       Proxy direct video URLs (avoids CORS/ad injection)
  ├─ /api/ts_status       TorrServer health
  ├─ /api/torrent_status  Torrent download status
  └─ /*                   Serves React/Vite frontend (e.g. SanuFlix) or static HTML

Frontend (React/Vite or static HTML)
  ├─ Movie search + grid with posters
  ├─ Movie detail page (synopsis, cast, rating)
  └─ Watch page with unified source grid + video player
```

## Free Sources (from Kodi addon RE)

### Tier 1 — Reliable
| Source | Type | How | Notes |
|--------|------|-----|-------|
| **TPB** (apibay.org) | Torrent | JSON API | Returns info hashes + seeders. Most reliable free source. |
| **BitSearch** | Torrent | HTML scrape | Secondary torrent source. |
| **2Embed** | Embed | iframe URL | Reliable embed, minimal ads. |
| **AutoEmbed** | Embed | iframe URL | Reliable embed, clean. |

### Tier 2 — Unreliable / Dead
| Source | Type | Why unreliable |
|--------|------|----------------|
| GDrivePlayer | Direct | API changes frequently |
| Ronemo | Direct | Site often down |
| DoodStream | Direct | Blocks server scraping, Cloudflare |
| VidCloud | Direct | HTML structure changes |
| StreamHide | Direct | Blocks scraping |
| Bing Search | Discovery | Random results, low hit rate |

### Tier 3 — Additional (from Kodi addons)
- VidSrc, EmbedSu — embeds with ads
- StreamWish, VidLink, Voxzer, FurHer — free hosts, often dead
- Gomo, Ronemo, LinkBin — similar free hosts

## TorrServer Integration

### Critical: Chicken-and-Egg Problem
TorrServer only starts downloading a torrent when it has an ACTIVE reader on the `/play/{hash}/{file_id}` endpoint. The frontend MUST set the stream URL immediately and let the browser connect, rather than polling for `files > 0` before setting the URL.

**WRONG** (polls, never starts):
```python
# Frontend polls for files > 0 - TorrServer never starts because no reader
# WRONG approach - chicken-and-egg
```

**RIGHT** (sets URL immediately):
```python
# Frontend sets stream URL right away
# Flask proxy:
# 1. Adds magnet to TorrServer
# 2. Immediately calls /play/{hash}/{fid}
# 3. Streams whatever comes back
# TorrServer sees reader, starts downloading
```

### Stream Endpoint Flow
1. Call `/api/add_magnet?hash=XXX` — adds magnet to DB (does NOT consume stream data)
2. Browser requests `/stream/XXX` — Flask calls TorrServer's `/play/XXX/1`
3. TorrServer connects to peers, downloads metadata, streams data
4. Flask proxies data to browser as `Content-Type: video/mp4`, `Content-Disposition: inline`

### Auto-Detecting Video File
When `file_stats` are available:
- Look for files ending in `.mp4`, `.mkv`, `.avi`, `.mov`, `.webm`, `.m4v`
- Pick the LARGEST video file (by `length` field)
- If no video match, use first file as fallback

If content-type is `text/*`, try file_id 2, then 1.

### Separate Magnet Add from Stream
**CRITICAL**: The magnet add and stream endpoints MUST be separate.
- `/api/add_magnet?hash=XXX` — just adds the magnet, returns JSON
- `/stream/XXX` — only called by the browser's `<video>` tag

If you `fetch(/stream/XXX)` from JavaScript to trigger the magnet add, the Python code consumes the stream data before the browser can use it.

### Torrent Status States
From TorrServer MatriX fork:
- `"Torrent in db"` — saved to DB, not active. Need a stream reader to activate.
- `"Torrent getting info"` — connecting to DHT/trackers
- `"Torrent working"` — downloading, streaming possible
- `stat: 0` = stored; `stat: 1` = getting info; `stat: 2` = working

## Embed Source Ads Problem

**You CANNOT block ads in embed iframes server-side.** The embed sites serve ads as part of their page. Sandbox attribute (`sandbox="allow-scripts"`) breaks the embed functionality.

**Why Android apps don't have ads:** They load the embed page in a hidden WebView, intercept network requests to extract the direct `.mp4`/`.m3u8` video URL, then play in a native `ExoPlayer` / `VideoView` — bypassing the ad HTML entirely. This requires a full headless browser server-side.

**Best options for ad-free:**
1. P2P torrents via TorrServer (self-hosted, zero ads)
2. A browser with ad blocking on the client device
3. Paid debrid service (Real-Debrid, Premiumize)

## Frontend Source Display

Show ALL sources in one unified grid (not separate sections):
- **Torrent** sources: `P2P 720s 1080p` (seeders + quality)
- **Direct** sources: `GDrive 1080p` (source + quality)
- **Embed** sources: `2Embed` (source name only)

Color-code by type: green for torrent/P2P, indigo for direct, gray for embed.
Show "Show all N sources" button if >6 sources.

## Embed Extractor — Priority Chain (Node.js Implementation)

When extracting m3u8/MP4 URLs from embed sites server-side, use a layered extractor registry with priority fallback. The unified implementation lives at `~/ultimate-scraper/src/providers/embed/`.

### Extractor Priority Chain (8 extractors registered)

```
Priority 100 — VidSrc AJAX API (calls /ajax/embed/episode/{dataId}/sources → RC4 decrypt)
Priority  90 — Filemoon (ECDSA P-256 attestation + AES-256-GCM decrypt)
Priority  90 — Rabbitstream/Megacloud (AES-CBC decrypt from player script)
Priority  80 — VidSrcNet (companion.js redirect, iframe follow, m3u8 regex)
Priority  80 — Streamtape (innerHTML = '...mp4' regex)
Priority  80 — VOE (video tag, iframe chain, redirect)
Priority  75 — SuperEmbed (multiembed.mov, no API key needed, free)
Priority   5 — Generic HTTP (brute-force m3u8/mp4 regex in HTML + scripts + iframes)
```

### Filemoon — ECDSA P-256 Attestation Flow

Filemoon is the hardest embed hoster to crack. It requires:
1. GET `/api/videos/{id}/embed/details` → `embed_frame_url`
2. POST `/api/videos/access/challenge` → `challenge_id` + `nonce`
3. Generate ECDSA P-256 keypair. Sign `nonce` with private key. Build JWK from public key.
4. POST `/api/videos/access/attest` with JWK + signature + browser fingerprint (UA, screen, languages, storage)
5. POST `/api/videos/{id}/embed/playback` with attestation token
6. AES-256-GCM decrypt `playback.payload` using composite key (`key_parts[0]` + `key_parts[1]`)

See `~/ultimate-scraper/src/providers/embed/filemoon.js` for full Node.js implementation.

### Rabbitstream/Megacloud — AES-CBC Decrypt

Find the player script (`evpHls`), extract `data`/`key`/`iv` from JS variables, AES-256-CBC decrypt to reveal m3u8 URL.

### VidSrc — Two Extraction Paths

1. **AJAX API path** (priority 100): Load embed page, extract `data-id` from anchor element, call `/ajax/embed/episode/{dataId}/sources` then `/ajax/embed/source/{sourceId}`, RC4 decrypt the result with key `'WXrUARXb1aDLaZjI'`.
2. **HTML-based path** (fallback): Look for `rc4('key', 'str')` calls in page JS, iframe redirects, direct m3u8 regex.

### SuperEmbed — Free Embed API (No API Key)

SuperEmbed/multiembed.mov provides embed players for any TMDB/IMDB ID:

```
Movie: https://multiembed.mov/?video_id={tmdbId}&tmdb=1
TV:    https://multiembed.mov/?video_id={tmdbId}&tmdb=1&s={season}&e={episode}
```

Also has a JSON API (`seapi.link`) returning streaming links in JSON format. Rate limited: 10 req/10s per IP. Results valid for 48 hours.

### Why Server-Side Extraction Fails
Most embed sites use this anti-bot pattern that no server-side scraper can fully bypass:
```
1. if(window===window.top) redirect    → blocks direct access, requires iframe
2. navigator.webdriver check           → detects headless automation
3. JavaScript-rendered video URL       → never in static HTML
4. Click-to-play gate                  → video URL requires user interaction
5. Cloudflare challenge                → requires real browser JS execution
```

### The Only Reliable Approaches
1. **Debrid services** ($3/mo) — instant cached torrents, zero anti-bot
2. **Browser-based** (Playwright + stealth) — heavy, ~50% success rate on modern sites
3. **P2P torrents via TorrServer** — free, works, takes 15-30s peer connection
4. **Android native WebView intercept** — the approach Cinema HD, BeeTV, TeaTV use (see `references/android-app-scraping-architecture.md`)

## Embed Proxy for CORS-Free Streaming

When the extracted video URL (m3u8/mp4) resides on a different domain than the app, browsers block the request due to CORS. Implement a proxy endpoint:

```
GET /api/embed/proxy?url=https://cdn.example.com/stream.m3u8&referer=https://source.com
```

For **m3u8 manifests**: fetch the manifest, rewrite relative segment URLs to proxied absolute URLs, return with `Content-Type: application/vnd.apple.mpegurl`.

For **mp4/ts segments**: stream the response through with `Content-Type` and `Accept-Ranges: bytes` headers forwarded.

See `~/ultimate-scraper/src/routes/proxy.routes.js` for a production implementation.

## Merged Scraper Stack

The complete system now uses a **single unified service** at `~/ultimate-scraper/` (port 3091) that combines torrent scraping + embed extraction + debrid integration in one Node.js Express app. The old three-service architecture (Flask + ultimate-scraper + free-scraper) has been consolidated.

### Unified Service (July 2026)

```
ultimate-scraper (:3091)
├── 11 torrent providers
│   ├── TPB (apibay JSON), YTS (JSON), BitSearch (HTML), 1337x (API+HTML fallback)
│   ├── EZTV (HTML table), Torznab (Jackett/Prowlarr)
│   └── Anime: NyaaSi, TokyoTosho, AniDex, SubsPlease, AnimeTosho
├── 6 embed extractors (8 registration entries, priority chain)
│   ├── VidSrc API (AJAX + RC4), Filemoon (ECDSA + AES-256-GCM)
│   ├── Rabbitstream (AES-CBC), Streamtape (regex), VOE (redirect)
│   └── SuperEmbed (multiembed.mov), Generic HTTP (brute force)
├── 4 debrid providers
│   ├── Real-Debrid (addMagnet → selectFiles → poll → unrestrict)
│   ├── AllDebrid (magnet/upload → status → unlock)
│   ├── Premiumize (directdl → transfer → poll)
│   └── TorBox (createtorrent → mylist → requestdl)
├── Health monitor (circuit breaker per provider)
│   ├── 3 failures = degraded, 5 failures = down
│   └── Exponential backoff: 30s → 60s → 120s → 240s → 300s
├── SQLite persistent cache (WAL mode, configurable TTL)
└── Stremio-compatible output: embed sources first, torrents second
```

### Unified Stream Output

The Stremio `/stream/:type/:id.json` endpoint returns **embed sources first** (instant, free, no debrid) followed by torrent results:

```json
{
  "streams": [
    { "name": "[EMBED] MultiEmbed", "url": "https://multiembed.mov/...", "quality": "1080p" },
    { "name": "[EMBED] VidSrc", "url": "https://vidsrc.to/embed/movie/tt0133093", "quality": "1080p" },
    { "name": "[TPB] The.Matrix.1999.1080p...", "infoHash": "abc123...", "seeders": 42, "cachedProviders": ["RD"] }
  ]
}
```

Embed sources with an `embedType: 'iframe'` flag should be rendered in a WebView/iframe. Torrent sources with `infoHash` should flow through debrid or TorrServer.

### Real-Debrid Cache Checking (addMagnet flow)

`/torrents/instantAvailability/{hash}` is **DEPRECATED** (error 37). Use addMagnet + selectFiles + info poll instead:

```javascript
// 1. Add magnet
const addResp = await axios.post(`${RD_API}/torrents/addMagnet`,
  `magnet=${encodeURIComponent(magnetLink)}`,
  { headers: { Authorization: `Bearer ${rdKey}` } }
);
const torrentId = addResp.data.id;

// 2. Select all files
await axios.post(`${RD_API}/torrents/selectFiles/${torrentId}`,
  'files=all', { headers: { ... } }
);

// 3. Poll info up to 3 times (4.5s max)
for (let i = 0; i < 3; i++) {
  await new Promise(r => setTimeout(r, 1500));
  const infoResp = await axios.get(`${RD_API}/torrents/info/${torrentId}`);
  if (infoResp.data.status === 'downloaded' || infoResp.data.status === 'waiting_files_selection') {
    return true; // Cached
  }
}

// 4. Clean up — delete the check torrent
await axios.post(`${RD_API}/torrents/delete/${torrentId}`);
```

### Health Monitor — Circuit Breaker Pattern

Per-provider circuit breaker with exponential backoff:

| Consecutive Failures | State | Cooldown |
|---------------------|-------|----------|
| 0-2 | Healthy | — |
| 3-4 | Degraded | 30s + 60s |
| 5-6 | Down | 120s + 240s |
| 7+ | Down | 300s max |

Implementation: `~/ultimate-scraper/src/services/healthMonitor.js`

### Multi-Debrid Integration (4 providers)

instantAvailability is **DEPRECATED** (error 37, mid-2024). The correct flow for all providers is addMagnet → poll → unrestrict.

| Provider | Resolve Endpoint | API Pattern |
|----------|-----------------|-------------|
| Real-Debrid | `/api/rd/resolve?hash=X` | addMagnet → selectFiles → poll info → unrestrict/link |
| AllDebrid | `/api/ad/resolve?hash=X` | magnet/upload → status poll → link/unlock |
| Premiumize | `/api/pm/resolve?hash=X` | directdl check (instant if cached) → transfer/create → poll → directdl |
| TorBox | `/api/tb/resolve?hash=X` | createtorrent → mylist poll → requestdl |
| **Auto** | `/api/debrid/resolve?hash=X` | Tries all enabled providers in order |

Config auto-detection: set `RD_API_KEY`, `AD_API_KEY`, `PM_API_KEY`, `TB_API_KEY` in `.env` to enable each.

### TorrServer Chicken-and-Egg Fix

**CRITICAL**: TorrServer only starts downloading a torrent when it has an ACTIVE reader on the `/play/{hash}/{file_id}` endpoint. The frontend MUST set the stream URL immediately rather than polling for `files > 0`.

**WRONG** (polls, never starts):
```javascript
// Poll for files > 0 — TorrServer never starts because no reader connected
// files stays at 0 forever → infinite loop
```

**RIGHT** (sets URL immediately):
```javascript
// 1. Call /api/add_magnet?hash=XXX to add magnet
// 2. Set stream URL right away: <video src="/stream/XXX">
// 3. Flask calls TorrServer's /play/XXX/1
// 4. TorrServer sees reader, starts downloading, serves data
// 5. Browser plays as data arrives
```

These apps DO NOT use Stremio addons (Torrentio, Comet). They ARE the scraper — everything is compiled into the APK.

### Flow
```
         ┌──────────────┐
         │  Torrent     │  1337x, TPB, YTS, EZTV,
         │  Site        │  TorrentGalaxy, Kickass,
         │  Scrapers    │  LimeTorrents (15-20 sites)
         └──────┬───────┘
                │ info hashes
                ▼
         ┌──────────────┐
         │  Real-Debrid │  POST /torrents/instantAvailability
         │  API Check   │  (or: addMagnet → selectFiles → poll → unrestrict)
         └──────┬───────┘
                │ HTTP link from RD CDN
                ▼
         ┌──────────────┐
         │  ExoPlayer   │  Direct playback from RD's servers
         └──────────────┘
```

### Why They Don't Need Stremio Addons
- **Built-in scrapers** — 15-20 torrent site scrapers compiled into the APK itself (Kotlin/Java)
- **Direct hash extraction** — scrape pages from 1337x, YTS, EZTV, TPB, etc., extract info hashes
- **RD API integration** — call `instantAvailability` (deprecated) or fallback to `addMagnet` → poll → `unrestrict`
- **Native player** — play RD's direct HTTP link in ExoPlayer/Media3, zero ads

### Non-Debrid Path
For users without debrid:
- Direct HTTP scrapers for free hosts (StreamTape, DoodStream, MixDrop, FileMoon)
- Same extractor pattern as our free-scraper, but running in **WebView** on device
- WebView executes JavaScript, bypasses Cloudflare naturally
- App intercepts network requests to capture direct video URLs
- Falls back to browser-based extraction (the user's real browser handles the anti-bot)

### Key Difference from Server-Side Scraping
| Approach | Runs where | Bypasses Cloudflare? | Success Rate |
|----------|-----------|---------------------|-------------|
| Server-side HTTP | Server | ❌ Blocked | ~10% |
| Playwright headless | Server | 🟡 Sometimes | ~50% |
| **WebView on device** | **User's phone** | **✅ Yes** | **~95%** |
| Native ExoPlayer + RD | Phone | ✅ Yes | 100% |

The WebView approach is the secret sauce — it runs on the user's actual device with a real browser engine,
so Cloudflare can't distinguish it from a normal user. The app loads the embed page in a hidden WebView,
intercepts ALL network requests, and captures the `.m3u8`/`.mp4` URLs. Then it stops the WebView and
plays the URL in the native player. This is impossible to replicate server-side without running a full
browser with a real user profile.

### Relation to Our System
- Our free-scraper's **Playwright extractor** is the server-side equivalent of the WebView approach
- The difference: Playwright runs on the server (IP reputation matters), WebView runs on the user's device
- For a truly reliable free solution, the extraction MUST happen on the client device, not the server

### See Also
- `references/android-app-scraping-architecture.md` — Detailed flow diagrams, RD API call sequence, scraper patterns extracted from decompiled APKs

## SanuFlix React Frontend Integration

Replace Flask Jinja2 templates with a React/Vite SPA for a better UI:

### Setup
```bash
git clone <sanuflix-repo>
cd sanuflix && npm install
# Set env vars in .env:
#   VITE_TMDB_API_KEY=<key>
npm run build  # Outputs to dist/
```

### Flask serves static build
```python
DIST = '/path/to/sanuflix/dist'
@app.route('/')
@app.route('/<path:path>')
def serve(path='index.html'):
    fp = os.path.join(DIST, path)
    if path and os.path.isfile(fp):
        return send_from_directory(DIST, path)
    return send_from_directory(DIST, 'index.html')
```

### Key Frontend Modifications
1. **Remove snowflake effects** — delete ChristmasSnow import, snow CSS animations
2. **Unified source grid** — show ALL source types (torrent + embed + direct) in one place
3. **Player**: `<video>` tag for direct URLs (mp4/m3u8), iframe for embed URLs
4. **P2P flow**: Show loading overlay for 8s, then set stream URL (don't poll for files)
5. **Source labels**: `P2P 720s 1080p` for torrents, `2Embed` for embeds, `GDrive` for direct
6. **"Show all N sources"** button when >6 sources

### Updating the Server Config
```typescript:src/config/servers.ts
// Only sources that actually work
export const STREAMING_SERVERS = [
  { id: '2embed', name: '2Embed', movieUrlTemplate: 'https://www.2embed.cc/embed/{tmdbId}' },
  { id: 'autoembed', name: 'AutoEmbed', ... },
  { id: 'vidsrc', name: 'VidSrc', ... },
  { id: 'embedsu', name: 'EmbedSu', ... },
]
```

### Pitfalls

#### P2P Streaming NOT Working
1. **Wrong file_id** — file 1 is often a subtitle/text file. Auto-detect video files by extension.
   - Check `file_stats` for video extensions (`.mp4`, `.mkv`, `.avi`, `.mov`, `.webm`, `.m4v`)
   - Pick the LARGEST video file
   - If content-type from TorrServer is `text/*`, try file_id 2 then 1
2. **Browser downloads instead of plays** — missing `Content-Disposition: inline` header. Flask must add this to the stream response.
3. **Torrent never starts (chicken-and-egg)** — TorrServer only downloads when it has an ACTIVE reader on `/play/{hash}/{file_id}`. The frontend MUST set the stream URL immediately (don't poll for `files > 0`).
4. **TPB hashes are old/dead** — many TPB results have no real seeders. Filter by `seeds > 0` but even then some are stale.
5. **TorrServer needs open ports** — on some networks, can't connect to DHT/trackers. Symptom: torrents stuck on "Torrent getting info" with 0 peers.
6. **Separate magnet add from stream** — NEVER `fetch(/stream/<hash>)` from JavaScript to trigger a magnet add. The Python proxy consumes the stream data before the browser can use it. Use a dedicated `/api/add_magnet?hash=XXX` endpoint instead.

### Embed Sources
1. **Sandbox breaks everything** — don't use sandbox for embed iframes. Mark problematic sources in the UI instead.
2. **Free embed hosts change URLs** — maintain a list of working domains and test them periodically.
3. **Cloudflare blocks scraping** — use browser User-Agent headers and handle 403 errors gracefully.
4. **Embed URLs cause ExoPlayer crash.** Passing an embed page URL (e.g. `https://multiembed.mov/?video_id=...`) as `Stream.url` makes ExoPlayer throw `UnrecognizedInputFormatException: None of the available extractors... could read the stream`. Fix: `url` must be `null` for embed sources in APK code. Store the real URL in `description` with an `embed://` prefix, and resolve via server proxy at playback time.
5. **Smart proxy auto-resolves embed pages.** `/api/embed/proxy` in the server detects embed URLs vs direct video URLs. If the input lacks a video extension (`.m3u8`, `.mp4`, `.ts`), it runs resolution first via the extractor registry, then proxies the result. APK can pass any URL and get back a playable stream.
6. **APK embed sources require server proxy.** Without `NEXSTREAM_SCRAPER_URL`, embed sources in the Android app are informational only — the server must be running for actual playback.

### General
1. **API rate limiting** — TMDB rate-limits at ~50 req/s. TPB (apibay.org) has no rate limit but may go down.
2. **SSL certificate issues** — use `ssl._create_unverified_context` for scraping, but prefer verified where possible.
3. **Unicode in torrent names** — always use `utf-8`, `errors='replace'` when decoding.
4. **React state timing** — `setActiveUrl` in React fires async. Use `useEffect` with dependency, not `setTimeout`.

### See Also
- `references/embed-extractor-techniques.md` — Detailed extraction flows for 8 embed hosters (ECDSA, AES-CBC, RC4, etc.)
- `references/free-source-providers.md` — Complete list of free video host domains from Scrubs V2 and other Kodi addons
- `references/torrserver-api.md` — TorrServer REST API reference (MatriX fork endpoints, streaming URLs, status codes)
- `references/scraper-techniques-research.md` — Research findings from 10+ scraper projects: Playwright headless, AJAX API extraction, RC4 decryption, anti-bot bypass techniques, and the reliability reality
- `references/android-app-scraping-architecture.md` — How Cinema HD / BeeTV / TeaTV scrape without Stremio addons
- `references/merged-scraper-architecture.md` — How Flask, nexstream-scraper, and free-scraper coexist + adding new scrapers + RD API flow + Torrentio provider list