# Ultimate Scraper Reference (v4.1)

Standalone multi-source torrent scraper + Stremio addon + multi-debrid. Independent of any app.

**Location:** `~/ultimate-scraper/`
**Port:** 3091
**Run:** `npm start`
**Dependencies:** express, axios, cors, dotenv, node-cache, cheerio (no SQLite, no Playwright in this service)

## Sources (10 scrapers)

| Source | API | Status | Type |
|--------|-----|--------|------|
| TPB | `apibay.org/q.php?q={title}&cat=0` | ✅ Always | Public torrent |
| Nyaa.si | `nyaa.si/?f=0&c=0_0&q=...` — cheerio table parse | ✅ Always | Anime torrent |
| AniDex | `anidex.info/api/?q=...` — JSON API | ✅ Always | Anime torrent |
| SubsPlease | `subsplease.org/api/?f=search&s=...` | 🟡 Sometimes | Anime torrent |
| AnimeTosho | `animetosho.org/search?q=...` — RSS HTML | 🟡 Sometimes | Anime torrent |
| YTS | `yts.mx/api/v2/list_movies.json` | 🟡 Intermittent | Public torrent |
| 1337x | `torrent-api-py.onrender.com` → direct HTML fallback | 🟡 Unstable | Public torrent |
| BitSearch | `bitsearch.to/search?q=...` | 🟡 Site changes | Public torrent |
| TokyoTosho | `tokyotosho.info/search.php?terms=...` — cheerio parse | 🟡 Site changes | Anime torrent |
| Torznab | Jackett/Prowlarr Torznab XML endpoint | 🟡 Requires config | Any indexer |

## Endpoints

```
GET /health                              → {"status":"ok","version":"4.1.0"}
GET /api/scrape?title=The+Matrix         → {"total":N,"streams":[...]}
GET /stream/movie/tt0133093.json         → Stremio stream protocol
GET /manifest.json                       → Stremio addon manifest
GET /config                              → Active scrapers + debrid status

# Multi-Debrid
GET /api/rd/resolve?hash=XXX             → Real-Debrid: addMagnet → selectFiles → poll → unrestrict
GET /api/ad/resolve?hash=XXX             → AllDebrid: magnet/upload → status poll → link/unlock
GET /api/pm/resolve?hash=XXX             → Premiumize: directdl check → transfer/create → poll → directdl
GET /api/tb/resolve?hash=XXX             → TorBox: createtorrent → mylist poll → requestdl
GET /api/debrid/resolve?hash=XXX         → Auto: tries all enabled providers in order

# Free embeds (via free-scraper :3092)
GET /api/resolve?url=https://...         → Playwright-based embed extraction
```

## Debrid API Patterns

### Real-Debrid
```
POST /torrents/addMagnet {magnet: "magnet:?xt=urn:btih:HASH&dn=video"}
POST /torrents/selectFiles/{id} {files: "all"}
GET /torrents/info/{id} [poll until status=downloaded]
POST /unrestrict/link {link: info.links[0]}
→ Returns {download, filename, size}
```

**instantAvailability is DEPRECATED (error 37)**. Do NOT use. No bulk cache-check is possible server-side anymore. The addMagnet→selectFiles→poll→unrestrict flow IS the resolve path. Cached torrents complete in ~2 seconds; uncached ones fail after 30s timeout.

### AllDebrid
```
GET /v4/magnet/upload?apikey=KEY&magnets[]=MAGNET
GET /v4/magnet/status?apikey=KEY&id=MAGNET_ID [poll until status=Ready]
GET /v4/link/unlock?apikey=KEY&link=STATUS.links[0].link
→ Returns {link, filename, size}
```
Agent parameter: use `ultimate-scraper`. AllDebrid has the fastest response for cached torrents (~1-2s).

### Premiumize
```
POST /api/transfer/directdl {apikey: KEY, magnet: MAGNET}
→ If cached: returns {content: [{link, name, size}]} instantly
→ If not: returns {status: "success", content: []}

POST /api/transfer/create {apikey: KEY, src: MAGNET}
GET /api/transfer/list?apikey=KEY [poll until status=finished]
POST /api/transfer/directdl {apikey: KEY, src: file_id}
→ Returns {content: [{link, name, size}]}
```
Premiumize is unique — `directdl` returns instantly for cached content with no magnet creation needed.

### TorBox
```
POST /v1/api/torrents/createtorrent {magnet, seed: 0, allow_zip: false, as_queued: false}
GET /v1/api/torrents/mylist?id=TORRENT_ID [poll until download_present=true]
GET /v1/api/torrents/requestdl?id=TORRENT_ID&token=KEY
→ Returns download URL
```
TorBox requires `Authorization: Bearer KEY` header. Resolve is slower than other providers (TorBox stores on their cloud first).

## Caching

In-memory cache (node-cache) wired into `scrapeAll()`. 10-minute TTL per query. Identical title searches return cached results in ~15ms on subsequent calls. Cache key: `scrape:{lowercase(title)}`.

## Key Files

```
ultimate-scraper/
  src/index.js            Express server (10 scraper list, multi-debrid routes)
  src/scrapers/ultimate.js  Merged scraper core (10 sources + 4 debrid resolve functions)
  src/cache.js            In-memory cache (wired into scrapeAll)
  src/health.js           Health endpoint (v4.1.0)
  src/config.js           Config layer (env + JSON + multi-debrid key detection)
  src/config/default-config.json  Full config: scrapers, debrid, RTN ranking, presets
  package.json            Standalone deps (cheerio added for Nyaa/TokyoTosho parsing)
  README.md               Full docs
```

## Nyaa.si Cheerio Parsing Pattern

The `scrapeNyaa` function uses cheerio to parse Nyaa's HTML table for structured metadata:
- Title: from `<td colspan="2">` last `<a>`'s `title` or text
- Hash: from magnet link in the `<td class="text-center">` column
- Seeders: from the 4th `<td class="text-center">` (eq(2))
- Size: from the 3rd `<td class="text-center">` (eq(1)) — parsed via `parseSize()`
- Category: from first column `<a title="...">`

Fallback: raw magnet regex extraction if cheerio parsing returns zero results (site structure change).

## Torznab Integration

Activate by setting in `.env`:
```
JACKETT_URL=http://localhost:9117
JACKETT_API_KEY=your_key_here
```
Scrapes `/api/v2.0/indexers/all/results/torznab/api` with `t=search&q=TITLE`. Parses XML for `<item>` blocks extracting infohash, title, seeders, size, and indexer name. Adds 500+ private/public trackers through Jackett/Prowlarr. Config auto-detects env vars — no code change needed.