# NexStream / Unspooled Scraper Architecture

## Overview

nexstream-scraper is a self-hosted Node.js Stremio-compatible addon that aggregates multiple torrent scrapers behind a single endpoint. It is debrid-agnostic — returns infoHash+magnet only. The Android app handles all debrid resolution client-side via DebridEnricher + user's connected debrid accounts.

## Location

`~/Unspooled/services/nexstream-scraper/`

## Stack

- Node.js 20+ (ES modules), Express, better-sqlite3
- Stremio addon-sdk for manifest format
- Docker-ready (multi-stage Dockerfile + docker-compose.yml)

## Scrapers (5, all run in parallel via Promise.allSettled)

| Scraper | Endpoint | Optional? | Max results |
|---------|----------|-----------|-------------|
| Torrentio | torrentio.strem.fun | No | 200 |
| Comet | comet.elfhosted.com | No | 200 |
| MediaFusion | mediafusion.elfhosted.com | No | 200 |
| Jackett | self-hosted (JACKETT_URL) | Yes | 200 |
| Zilean DMM | self-hosted (ZILEAN_URL) | Yes | 200 |

10s timeout per scraper. Failed/tiemout scrapers return empty — never block others.

## Pipeline

1. Request → `/stream/:type/:id.json?season=N&episode=N`
2. Parallel fan-out to all 5 scrapers
3. Dedup by infoHash (Set-based)
4. Sort by seeders descending
5. 2-tier cache: L1 NodeCache (30min TTL) + L2 SQLite (24h TTL)
6. Return Stremio-formatted stream objects (infoHash, magnet, size, seeders, tracker)

## Debrid Modules (DEAD CODE)

`src/debrid/realdebrid.js`, `torbox.js`, `alldebrid.js`, `premiumize.js` are fully implemented API clients with rate limiters, cache checking, magnet adding, and link unrestriction — but NOT imported anywhere. They exist from an earlier design where the server would handle debrid resolution. The current design is client-side only.

## App Integration

- Registered as system addon `nexstream_scraper` in SystemAddonCatalog.kt
- Enabled by default, priority 15
- URL from `BuildConfig.NEXSTREAM_SCRAPER_URL` (normally http://192.168.1.50:3091)
- Old addons RETIRED: torrentio, comet, mediafusion, jackettio, knightcrawler, annatar (in REMOVED_SYSTEM_IDS)
- These all proxied through nexstream-scraper — NO client-side fallback

## Deployment Status

**Built but NOT deployed.** No Docker image built, no container running. Never been health-checked in production.

## Reference: SEL-Filtering-and-Sorting

Tamtaro's AIOStreams template set at `~/SEL-Filtering-and-Sorting/`. Contains:
- Tamtaro-All-Templates-for-AIOStreams.json (the main SEL template)
- AIOStreams Templates/ (complete + partial setup templates)
- AIOStreams-SyncedURLs/ (community-maintained exclude/include/passthrough lists)
- AIOMetadata Configs/ (AIOMetadata configs with/without anime)

SEL concepts to port to our scraper:
- **ReSE** (Ranked Stream Expressions) — score streams by quality: REMUX=100, 4K=90, 1080p=50, HDR=+10, DV=+15
- **ESE** (Exclude Stream Expressions) — drop CAM, TS, HC, low-quality releases
- **ISE** (Include Stream Expressions) — always keep certain releases
- **PSE** (Passthrough Stream Expressions) — bypass filters for specific patterns
- **Synced URLs** — auto-updated community blocklists