# Scraper Pipeline Architecture (Node.js)

## Overview

A config-driven, high-performance Stremio addon scraper that aggregates multiple upstream sources (Torrentio, Comet, MediaFusion, Jackett, Zilean), enriches with debrid cache status, filters/scores by quality, and caches aggressively.

## Pipeline Stages

```
Request → Cache Check → Parallel Scrape → Normalize → DSU Dedup
    → Debrid Enrich → Adaptive Filter → Quality Rank → Limit → Return
```

### 1. Parallel Scrape
- All enabled scrapers fire simultaneously via `Promise.allSettled`
- Per-scraper timeout via `Promise.race` (default 10s)
- Circuit breaker per scraper — auto-disables after N failures, resets after timeout
- Max 200 results per scraper, capped at 500 total

### 2. DSU Dedup (Disjoint Set Union)
3-pass dedup with resolution awareness:
- **Pass 1:** infoHash exact match
- **Pass 2:** Same resolution tier + cleaned title match
- **Never merges different resolutions** (4K stays separate from 1080p)
- Picks best per group: cached > higher res > more seeders > larger size

### 3. Debrid Enrichment
- Checks all 4 providers (RD/AD/PM/TB) in parallel
- Debrid availability cache: stores per-hash-per-provider result with 5min TTL
- Skips API call for hashes already in cache
- Tags streams: `cached: true/false`, `cachedProviders: [provider names]`
- Two modes:
  - `cachedOnly: false, uncachedMode: keep` — show all, tag cached
  - `cachedOnly: true` — return only cached
  - `cachedOnly: false, uncachedMode: exclude` — strict debrid-only

### 4. Quality Detection
Filename-based regex detection of:
- **Resolution:** 4320p/2160p/1440p/1080p/720p/480p/360p
- **Source:** REMUX, BluRay, WEB-DL, HDTV
- **Codec:** HEVC/x265, AV1
- **HDR:** Dolby Vision (Profile 5 & 7), HDR10/HDR10+
- **Audio:** Atmos, TrueHD, DTS-HD
- **Quality tags:** CAM, TS, HC, SAMPLE, 3D/SBS

### 5. Adaptive Filtering
- If 50+ high-quality streams (4K/REMUX/FHD) exist, drop SD/unknown/low-quality
- Standard exclusions: CAM, TS, TC, TeleSync, Telecine, HDTS, HC, 3D, SBS, samples
- Configurable include/require/exclude regex patterns
- Configurable per-language exclusion

### 6. Quality Ranking
Weighted scoring algorithm:

| Factor | Weight |
|--------|--------|
| Resolution | 2160p=100, 1080p=50, 720p=25, 480p=10 |
| REMUX | +40 |
| Dolby Vision | +20 |
| HDR | +15 |
| Atmos | +10 |
| TrueHD | +8 |
| DTS-HD | +6 |
| AV1 | +10 |
| HEVC | +5 |
| Cached | +50 |
| Seeders | min(seeders, 500) × 0.1 |
| Source reliability | Torrentio +5, Comet +3, MediaFusion +3 |
| CAM/TS/HC | -200/-150/-100 |

## Config-Driven Architecture

```json
{
  "scrapers": { "torrentio": { "enabled": true, "baseUrl": "...", "timeout": 10000 } },
  "debrid": { "realdebrid": { "enabled": true, "rateLimit": 60 } },
  "filtering": { "mode": "smart", "excludeQualities": ["CAM","TS","HC"] },
  "ranking": { "resolutionWeights": { "2160p": 100 }, "qualityBonuses": { "REMUX": 40 } },
  "limits": { "maxTotalStreams": 500, "cachedOnly": false },
  "performance": { "parallelDebridCheck": true, "circuitBreakerThreshold": 3 }
}
```

The `apiKey` fields in the JSON config are placeholder strings — actual keys come from env vars via config.js.

## Performance Optimizations

| Technique | Impact |
|-----------|--------|
| HTTP keep-alive pooling (shared http.Agent) | 3x throughput on repeated scrapes |
| L1 in-memory + L2 SQLite cache | 0ms cache hit vs 4.3s live scrape |
| Debrid availability cache (5min TTL) | Eliminates re-checking same hash across movies |
| Startup connection pre-warming | Eliminates cold-start TCP/TLS latency |
| Batch debrid hash checking | Single API call per provider for all hashes |
| Circuit breaker per scraper | Prevents hanging on dead endpoints |

## Key Dependencies

- **axios** with keep-alive agent (shared instance via `http-client.js`)
- **better-sqlite3** for L2 cache (WAL mode, auto-vacuum)
- **node-cache** for L1 in-memory cache
- **express** + **cors** for HTTP server

## Docker Deployment

```dockerfile
FROM node:20-alpine
# Multi-stage build with better-sqlite3 native addon
# Port 3091, healthcheck on /health
```

```yaml
services:
  nexstream-scraper:
    build: .
    ports: ["3091:3091"]
    volumes: ["scraper-data:/app/src/data"]
    env_file: .env
    restart: unless-stopped
```