# nexstream-scraper Architecture Reference

## Overview
Self-hosted Node.js Stremio addon (port 3091) that aggregates multiple torrent scrapers, checks debrid cache status, filters/ranks results, and returns Stremio-formatted streams. Debrid-agnostic — the Android app handles debrid URL resolution client-side with the user's own API keys.

## Pipeline

```
Request → /stream/:type/:id.json?season=N&episode=N
   │
   ├── PARALLEL SCRAPE (Promise.allSettled, per-scraper timeout)
   │   ├── Torrentio (torrentio.strem.fun)
   │   ├── Comet (comet.elfhosted.com)
   │   ├── MediaFusion (mediafusion.elfhosted.com)
   │   ├── Jackett (optional, self-hosted)
   │   └── Zilean (optional, self-hosted)
   │
   ├── NORMALIZE → standard Stream object per result
   ├── DEDUPLICATE — resolution-aware DSU (infoHash exact + normalized title)
   ├── DEBRID ENRICH — check cached status on all configured providers (parallel)
   ├── FILTER — exclude CAM/TS/HC/samples, min resolution, regex patterns
   ├── RANK — score = resolution_weight + quality_bonus + cached_bonus + seeders_bonus
   ├── LIMIT — top N by score
   └── CACHE → L1 memory (30min) + L2 SQLite (24h), 2ms subsequent hits
```

## Key Design Decisions

### Debrid-agnostic
Server returns infoHash + magnet only. The Android app's `DebridService` + `DebridEnricher` checks cache and resolves URLs client-side using the user's connected debrid accounts. No debrid credentials on the server.

### Server-side debrid cache checking (optional)
`src/debrid/` modules exist for RD/AD/PM/TB with rate limiters, cache checking, magnet adding, and link unrestriction. Wire by setting API keys in `.env` or `config/default-config.json`. When enabled, each stream gets a `cached` flag and `cachedProviders` list.

### Config-driven system
`src/config/default-config.json` controls all behavior — scrapers, debrid, caching, filtering, ranking, limits, performance. Ported from AIOStreams template structure. Env vars override JSON values.

### Circuit breakers
Per-scraper circuit breakers trip after 3 consecutive failures. Open for 5 minutes before retrying. Prevents a dead upstream addon from blocking every request.

### Streaming response support
`?debug=true` query param returns pipeline stats alongside results. `?cachedOnly=true` filters to cached-only. `?minResolution=1080p` overrides filter.

## Quality Detection (ports from Android app's ScraperQualityDetector)

Regex-based detection covering: resolution (8K→360p), source (REMUX, BluRay, WEB-DL, HDTV), codec (HEVC, AV1), HDR (DV, HDR10+, HDR), audio (Atmos, TrueHD, DTS-HD), quality tags (CAM, TS, HC, Sample, 3D).

## Ranking Weights

| Factor | Default Weight | Notes |
|--------|---------------|-------|
| 4K REMUX | 140 (100+40) | Resolution weight + REMUX bonus |
| 4K DV | 120 | Resolution + DV bonus |
| 4K HDR | 115 | Resolution + HDR bonus |
| 1080p REMUX | 90 | |
| Cached bonus | +50 | Applied per-stream |
| CAM penalty | -200 | Always excluded by filter |
| Seeders | 0.1× | Up to 50 bonus for 500 seeders |

## Caching Strategy

- **L1**: `node-cache` — in-memory, 30min TTL, 10K max keys
- **L2**: `better-sqlite3` — persistent, 24h TTL, auto-cleanup every 5min
- WAL mode + auto_vacuum for performance
- Cache keys: `stream:{type}:{id}` or `stream:{type}:{id}:{season}:{episode}`

## Docker Deployment

```dockerfile
Multi-stage build: node:20-alpine
- Builder: apk add python3 make g++ → npm install --production=false
- Runtime: apk add python3 → copy node_modules + src
- Non-root user (nexstream, uid 1001)
- Healthcheck: wget --spider /health every 30s
```

Run: `docker run -d --name nexstream-scraper -p 3091:3091 nexstream-scraper:v2`

## API Endpoints

| Route | Description |
|-------|-------------|
| `GET /stream/:type/:id.json` | Main pipeline — filtered, ranked, cached |
| `GET /stream/raw/:type/:id.json` | Raw output with pipeline stats |
| `GET /health` | Full pipeline health + cache stats |
| `GET /config` | Sanitized config (no keys exposed) |
| `POST /cache/clear` | Clear all cached results |

## Related
- Tamtaro's SEL-Filtering-and-Sorting repo — community AIOStreams config templates for reference
- AIOStreams — the reference implementation for SEL-based filtering and ranking
- Comet — example of parallel debrid + scrape pattern