# Merged Scraper Architecture (v4.1)

How ultimate-scraper (Node.js), free-scraper (Node.js), and Flask backend (Python) coexist. Nexstream-scraper was merged into ultimate-scraper — it no longer exists as a separate service.

## Service Stack

```
Flask Backend (:8800)              — Frontend + TMDB + TPB + embeds + TorrServer proxy
ultimate-scraper (:3091)            — 10 torrent scrapers + multi-debrid + Stremio protocol + cache
free-scraper (:3092)                — Playwright-based embed URL extraction
```

## Adding a New Scraper to `ultimate-scraper`

Create a function in `src/scrapers/ultimate.js` following the existing pattern:

```js
export async function scrapeMYSOURCE(title) {
  try {
    const { data } = await http.get(`https://mysource.example/search?q=${encodeURIComponent(title)}`);
    // Parse results, return array of:
    return data.map(item => buildResult(
      item.hash.toLowerCase(),
      item.name,
      item.seeders || 0,
      item.size || 0,
      'MySource',
      detectQuality(item.name),
    ));
  } catch (err) {
    log('MySource', err.message);  // Always log errors
    return [];
  }
}
```

Then add to the `scrapers` array in `scrapeAll()`:

```js
const scrapers = [
  { name: 'TPB', fn: scrapeTPB },
  { name: 'MySource', fn: scrapeMYSOURCE },
  // ...
];
```

**Must-dos**: use `log()` for error logging, `buildResult()` for output normalization, and `Promise.allSettled` pattern (no single scraper failure blocks others). Scraper results auto-dedup by infoHash, sorted by seeders descending, and cached for 10 minutes.

## Debrid API Flows (All 4 Providers)

### Real-Debrid
```
POST /torrents/addMagnet     → {magnet: "magnet:?xt=urn:btih:HASH&dn=video"}
POST /torrents/selectFiles/{id} → {files: "all"}
GET  /torrents/info/{id}      → poll until status=downloaded
POST /unrestrict/link         → {link: info.links[0]}
Return: {url: download, filename, size}
```
**instantAvailability is DEPRECATED (error 37)** as of mid-2024. No server-side bulk cache-check exists.

### AllDebrid
```
GET /v4/magnet/upload?apikey=KEY&magnets[]=MAGNET&agent=ultimate-scraper
GET /v4/magnet/status?apikey=KEY&id=MAG_ID   → poll until status=Ready
GET /v4/link/unlock?apikey=KEY&link=STATUS_LINKS[0].link
Return: {link, filename, size}
```

### Premiumize
```
POST /api/transfer/directdl    → if cached: instant {content: [{link, name}]}
POST /api/transfer/create      → if not cached: create transfer
GET  /api/transfer/list         → poll until status=finished
POST /api/transfer/directdl    → unlock
Return: {link: content[0].link, filename, size}
```

### TorBox
```
POST /v1/api/torrents/createtorrent → {magnet, seed:0, allow_zip:false}
GET  /v1/api/torrents/mylist?id=X   → poll until download_present=true
GET  /v1/api/torrents/requestdl     → get download URL
Return: {url, filename, size}
```

## Endpoints (Ultimate Scraper v4.1)

```
GET /api/scrape?title=The+Matrix     → {"total": N, "streams": [...]}  (10 scrapers, cached)
GET /stream/movie/tt0133093.json     → Stremio stream protocol
GET /manifest.json                   → Stremio addon manifest  
GET /api/rd/resolve?hash=XXX         → Real-Debrid resolve
GET /api/ad/resolve?hash=XXX         → AllDebrid resolve
GET /api/pm/resolve?hash=XXX         → Premiumize resolve
GET /api/tb/resolve?hash=XXX         → TorBox resolve
GET /api/debrid/resolve?hash=XXX     → Auto (tries all enabled in order)
GET /api/resolve?url=https://...     → Free embed extraction (via free-scraper :3092)
GET /health                          → {"status":"ok","version":"4.1.0"}
GET /config                          → Active scrapers + debrid status
```

No `/api/ultimate` or `rdKey` query param — instantAvailability was removed.

## Torrentio Provider List (for reference)

Sites Torrentio scrapes:
- YTS, EZTV, RARBG (dead), 1337x, TPB, KickassTorrents
- TorrentGalaxy, MagnetDL, HorribleSubs, NyaaSi, TokyoTosho
- AniDex, Rutor, Rutracker, Comando, BluDV, Torrent9
- ilCorSaRoNeRo, MejorTorrent, Wolfmax4k, Cinecalidad

Most are behind Cloudflare and only work server-side through FlareSolverr or WebView-based scraping.