# nexstream-scraper — Self-Hosted Stremio Scraper

## Overview

A Node.js Stremio-compatible addon that proxies 5 scraper backends behind a single endpoint. Runs on port 3091. Debrid-agnostic — returns infoHash+magnet only; the Android TV app handles debrid resolution client-side with user's own API keys.

**Location:** `~/Unspooled/services/nexstream-scraper/`
**Registered in app as:** `nexstream_scraper` system addon (enabled by default, priority 15)
**App URL source:** `BuildConfig.NEXSTREAM_SCRAPER_URL` → `http://192.168.1.50:3091`

## Flow

```
Client (Android TV) → GET /stream/:type/:id.json?season=N&episode=N
  │
  ├─ Torrentio    → torrentio.strem.fun/stream/:type/:id.json    (max 200)
  ├─ Comet        → comet.elfhosted.com/stream/:type/:id.json    (max 200)
  ├─ MediaFusion  → mediafusion.elfhosted.com/stream/:type/:id.json (max 200)
  ├─ Jackett      → /api/v2.0/indexers/all/results               (optional, self-hosted)
  └─ Zilean       → /dmm/filtered?query=                         (optional, self-hosted)
  │
  └─ Promise.allSettled → dedup by infoHash → sort by seeders → cache → return
```

## Source Files

| File | Purpose |
|------|---------|
| `src/index.js` | Express server (port 3091), middleware, route mounting, startup banner |
| `src/config.js` | All config from .env (URLs, timeouts, cache TTLs) |
| `src/addon.js` | Stremio manifest builder (`com.nexstream.scraper`, resources: stream+catalog, types: movie+series) |
| `src/routes/stream.js` | Core handler: parallel fan-out, dedup, caching, Stremio-formatted response |
| `src/cache.js` | 2-tier cache: L1 in-memory (NodeCache, 30min TTL) + L2 SQLite (better-sqlite3, 24h TTL) |
| `src/health.js` | Health endpoint with cache stats, scraper status, uptime |
| `src/scrapers/torrentio.js` | Calls Torrentio addon API, parses emoji-formatted metadata (👤 seeders 💾 size ⚙️ tracker) |
| `src/scrapers/comet.js` | Calls Comet addon API, parses description field for metadata |
| `src/scrapers/mediafusion.js` | Calls MediaFusion addon API, same parsing pattern |
| `src/scrapers/jackett.js` | Jackett/Torznab API (JSON + XML fallback), needs `JACKETT_URL` + `JACKETT_API_KEY` |
| `src/scrapers/zilean.js` | Zilean DMM search API, needs `ZILEAN_URL` |
| `src/debrid/realdebrid.js` | Not used — debrid is client-side. Present from earlier architecture iteration. |
| `src/debrid/alldebrid.js` | Not used |
| `src/debrid/premiumize.js` | Not used |
| `src/debrid/torbox.js` | Not used |

## Scraping Details

### Parallel Execution
- All 5 scrapers fire simultaneously via `Promise.allSettled`
- `withTimeout()` wrapper enforces per-scraper timeout (default 10s, configurable via `SCRAPER_TIMEOUT`)
- Failed/timed-out scrapers return empty array — never block others
- `Promise.allSettled` vs `Promise.all`: one scraper failure never rejects the whole batch

### Deduplication
- Simple Set-based dedup by `infoHash.toLowerCase()`
- Results without infoHash are kept (rare edge case)
- The Android app re-deduplicates with its own 4-pass DSU algorithm (StreamDeduplicator) — so double dedup is harmless
- Max 200 results per scraper, ~1000 max before dedup

### Caching
- **L1 (in-memory):** NodeCache, 30min TTL, 10K max keys
- **L2 (persistent):** SQLite with WAL mode, auto-vacuum, 24h TTL
- On L2 hit, repopulates L1 for faster subsequent access
- L2 auto-cleanup runs every 5 minutes (deletes expired entries)
- Cache key format: `stream:{type}:{id}` (e.g. `stream:movie:tt0133093`)

## Deployment

### Docker (not yet deployed)
- Dockerfile: multi-stage (node:20-alpine builder → node:20-alpine production), non-root user `nexstream:nexstream`
- docker-compose.yml: port 3091, `.env` file, persistent SQLite volume, healthcheck, bridge network
- Healthcheck: `GET /health` every 30s, 10s start period, 3 retries
- **Current status:** NOT deployed — code is ready, Docker image not built, container not running

### Manual (dev)
```bash
cd ~/Unspooled/services/nexstream-scraper
cp .env.example .env  # edit as needed
npm start              # runs on port 3091
```

## Configuration (.env)

| Key | Default | Notes |
|-----|---------|-------|
| `PORT` | 3091 | |
| `TORRENTIO_URL` | `https://torrentio.strem.fun` | |
| `COMET_URL` | `https://comet.elfhosted.com` | |
| `MEDIAFUSION_URL` | `https://mediafusion.elfhosted.com` | |
| `JACKETT_URL` | (empty) | Self-hosted only |
| `JACKETT_API_KEY` | (empty) | |
| `ZILEAN_URL` | (empty) | Self-hosted DMM |
| `CACHE_L1_TTL` | 1800 (30min) | In-memory TTL in seconds |
| `CACHE_L2_TTL` | 86400 (24h) | SQLite TTL in seconds |
| `SCRAPER_TIMEOUT` | 10000 (10s) | Per-scraper timeout in ms |
| `RD_API_KEY` etc. | (empty) | **Not used** — held over from earlier architecture |

**Note:** `RD_API_KEY`, `AD_API_KEY`, `PM_API_KEY`, `TB_API_KEY` in `.env.example` are NOT used by any scraper. Debrid resolution is entirely client-side in the Android app. These were from an earlier architecture iteration where debrid was handled server-side.

## Quality Assessment vs. Direct Addon Calls

| Aspect | Direct Calls | nexstream-scraper |
|--------|-------------|-------------------|
| **Latency** | TV → internet (parallel) | TV → server (~1ms) → internet (parallel). Negligible diff on LAN |
| **Caching** | None | 30min L1 + 24h L2 — major win for repeat content |
| **Dedup** | 4-pass DSU client-side | Simple Set-based (app re-dedups anyway) |
| **Jackett/Zilean** | Not available | Available (needs self-hosted setup) |
| **Rate limits** | Each addon provider rate-limits independently | Single server IP shares rate limit pool |
| **Resilience** | One addon down → others still work | Server down → zero streams (single point of failure) |
| **Quality detection** | App does it | App still does it (server passes raw titles) |
| **Debrid enrichment** | App does it | App still does it |

## Pitfalls

1. **Single point of failure** — all old addon IDs are in `REMOVED_SYSTEM_IDS`. If nexstream-scraper isn't running, the app returns zero streams with no client-side fallback.
2. **Scraper not deployed** — as of 2026-06-01, Docker image is not built, container not running. Code is complete but idle.
3. **Debrid API keys in .env are dead config** — `RD_API_KEY`, `AD_API_KEY`, `PM_API_KEY`, `TB_API_KEY` fields exist but no scraper reads them. They're from the earlier server-side debrid architecture that was abandoned.
4. **No server-side intelligence** — the server doesn't check debrid cache status, doesn't filter CAM/TS releases, doesn't rank by quality. All that happens client-side after data arrives.
5. **Jackett and Zilean empty by default** — without self-hosted instances, the scraper is just a proxy to 3 public addons (Torrentio, Comet, MediaFusion) that the app could call directly.