# NexStream Debrid-Agnostic Scraper Architecture

*Added 2026-06-01 — replaces the old in-app scraper pipeline*

## Why This Exists

The old architecture (documented in `debridstream-scraper-pipeline.md`) had scrapers running inside the Android app, calling addon endpoints directly. AIOStreams was the primary debrid engine but was unreliable and had broken configuration. Torrentio was a fallback but had frequent 598 errors.

**New architecture:** A self-hosted Node.js scraper on the local network aggregates 5 torrent sources. It returns only torrent metadata (infoHash + magnet). The Android app handles all debrid cache checking and URL resolution client-side using the end user's own API keys stored in Settings.

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│  nexstream-scraper (Node.js, port 3091, debian-ai:3091)        │
│                                                                 │
│  GET /stream/:type/:id.json                                     │
│                                                                 │
│  Parallel scrapers (Promise.allSettled):                        │
│    ├─ Torrentio   → torrentio.strem.fun                         │
│    ├─ Comet       → comet.elfhosted.com                         │
│    ├─ MediaFusion → mediafusion.elfhosted.com                   │
│    ├─ Jackett     → self-hosted (configurable)                  │
│    └─ Zilean      → self-hosted (configurable)                  │
│                                                                 │
│  Returns: [{ infoHash, magnet, name, seeders, size, tracker }]  │
│  NO debrid keys on the server. NO isCached in response.         │
└─────────────────────────────────────────────────────────────────┘
         │
         │ Stremio-formatted JSON
         ▼
┌─────────────────────────────────────────────────────────────────┐
│  Unspooled Android App                                          │
│                                                                 │
│  StremioAddonClient → calls nexstream_scraper system addon      │
│         │                                                       │
│         ▼                                                       │
│  DebridEnricher.enrich()                                        │
│    ├─ Gets user's API keys from DebridSettings (DataStore)      │
│    ├─ Calls debrid APIs directly (RD/AD/PM/TB)                  │
│    ├─ Checks instantAvailability for all infoHashes             │
│    └─ Marks streams as cached + which providers have them       │
│         │                                                       │
│         ▼                                                       │
│  PlaybackOrchestrator.prepare()                                 │
│    ├─ Ranks streams (cached first, then seeders)                │
│    ├─ Resolves magnet → HTTP URL via DebridService              │
│    │    • Add magnet to debrid → torrent ID                     │
│    │    • Get file list → pick best file                        │
│    │    • Get unrestricted download URL                         │
│    └─ Validates URL is playable                                 │
│         │                                                       │
│         ▼                                                       │
│  Player (Media3 ExoPlayer) ← HTTP streaming URL from CDN        │
└─────────────────────────────────────────────────────────────────┘
```

## Key Design Decisions

1. **Server has zero debrid credentials.** The scraper only aggregates torrent metadata. All debrid API calls happen on the user's device with their own keys.

2. **`notWebReady: true` in behaviorHints.** The Stremio protocol flag tells clients "don't play this URL directly — resolve through debrid first."

3. **App-side enrichment is authoritative.** The scraper used to return `isCached` based on server-side `.env` keys — this was wrong for the end user. Now `DebridEnricher` is the single source of truth for cache status.

4. **Magnet URI as the stream URL.** Each stream object has `url: "magnet:?xt=urn:btih:..."` — the app recognizes this and routes through debrid resolution.

## Files

| File | Purpose | LOC |
|------|---------|-----|
| `services/nexstream-scraper/src/index.js` | Express server, routes | 95 |
| `services/nexstream-scraper/src/routes/stream.js` | Stream handler (scrape + dedup) | 170 |
| `services/nexstream-scraper/src/config.js` | Env-based config (no debrid keys) | 40 |
| `services/nexstream-scraper/src/health.js` | Health/manifest endpoints | 38 |
| `services/nexstream-scraper/src/cache.js` | L1 (memory) + L2 (SQLite) cache | — |
| `services/nexstream-scraper/src/scrapers/*.js` | Per-source scrapers (5 files) | ~80 each |
| `app/.../data/addon/SystemAddonCatalog.kt` | `nexstream_scraper` seed def (priority 15) | — |
| `app/.../data/debrid/DebridService.kt` | Multi-provider debrid client | 404 |
| `app/.../data/addon/DebridEnricher.kt` | Cache check + stream enrichment | 66 |
| `app/.../domain/playback/PlaybackOrchestrator.kt` | Stream ranking + URL resolution | 269 |

## Running the Scraper

```bash
cd ~/Unspooled/services/nexstream-scraper
npm install
PORT=3091 node src/index.js
```

Health check: `curl http://localhost:3091/health`
Test streams: `curl http://localhost:3091/stream/movie/tt0111161.json`

## Docker

```bash
cd ~/Unspooled/services/nexstream-scraper
docker compose up -d
```

## Adding a New Scraper Source

1. Create `src/scrapers/newsource.js` with `export async function scrape(type, imdbId, opts)`
2. Import and add to `Promise.allSettled` in `stream.js`
3. Add config entry for URL in `config.js`

## Pitfalls

- **Don't add debrid cache checking to the scraper.** It's tempting to check `instantAvailability` server-side, but the server doesn't have the user's keys. Let the app handle it.
- **Don't return `isCached` from the scraper.** The app ignores it and runs its own check. Returning it is misleading.
- **Don't hardcode scraper URLs in the Android app.** They live in `nexstream-scraper/src/config.js` so they can be changed without an app update.
- **The scraper manifest URL is in `BuildConfig.NEXSTREAM_SCRAPER_URL`**. Set this per-environment: `http://192.168.1.50:3091` for LAN, or a Tailscale IP for remote.