# Config-Driven Stream Scraper Pipeline

**Source:** DebridStream FireTV APK (v3.3.1)
**Location:** `app/src/main/java/com/unspooled/app/data/scraper/`
**Files:** 7 files, ~500 LOC total

## Why it exists

Torrentio's public instance (`torrentio.strem.fun`) is frequently unreliable (598 errors, DNS issues, format changes). The old approach embedded debrid tokens into Torrentio URLs and had no fallback.

DebridStream doesn't use Torrentio at all. They use a **config-driven scraping pipeline** where:
1. Addons are defined as config objects (URL template + parse keys)
2. The engine calls all addons in parallel
3. Responses are parsed using the config's field mappings
4. Results are routed through connected debrid resolvers

This decouples us from any single addon's availability.

## Architecture (mirrors DebridStream 1:1)

```
DebridStream Class     →  Our File
────────────────────────    ────────────────────────
U2.c (ScraperAddon)       →  ScraperAddonConfig.kt
U2.h (ScraperStreamItem)  →  ScraperLink.kt
U2.g (ScraperResult)      →  ScraperResult (in ScraperLink.kt)
U2.f (JSON parser)        →  ScraperResponseParser.kt
U2.e (HTTP caller)        →  StreamScraperEngine.kt (scrapeAddon)
N2.q (orchestrator)       →  StreamScraperEngine.kt (scrape)
N2.q (debrid dispatch)    →  DebridRouter.kt
N2.e (CachedLinkResult)   →  ScraperQualityDetector.kt
C1125i (quality regex)    →  ScraperQualityDetector.kt (regex patterns)
C1123g (FilterOptions)    →  (future: QualityFlags-based filtering)
```

## Data Flow

```
ScraperAddonConfig[6] → URL with %searchPattern → HTTP GET (parallel)
  │
  ├─ AIOStreams   → aiostreams.elfhosted.com
  ├─ Comet        → comet.elfhosted.com
  ├─ MediaFusion  → mediafusion.elfhosted.com
  ├─ Jackettio    → jackettio.elfhosted.com
  ├─ Torrentio    → torrentio.strem.fun
  └─ PeerFlix     → peerflix.elfhosted.com
       │
  ScraperResponseParser.parse(body, config)
    Uses: jsonResultsKey, itemType, httpUrlTargetKey, sizeAttr
    → List<ScraperLink>(infoHash, directUrl, title, videoSize)
       │
  ScraperQualityDetector.detect(title, filename)
    Regex matching: 4K, 1080p, 720p, REMUX, HEVC, Atmos,
                    TrueHD, HDR, DV P5, DV P7, DV, DTS-HD,
                    AV1, 3D, CAM, Sample
    → QualityFlags (is4K, isFHD, isHevc, isHDR, isDV, ...)
    → Enriches stream title with "[4K REMUX DV HEVC Atmos]"
       │
  DebridRouter.route(link)
    ├─ infoHash → checkCachePerProvider → resolveStream
    ├─ directUrl → resolveStream (unrestrict)
    └─ plain HTTP → return directly
       │
  deduplicated Stream[] (by url/infoHash)
```

## DebridStream Addon Config Format

Each addon in DebridStream's Firestore has these fields (mirrored in ScraperAddonConfig):

| Field | Type | Default | Purpose |
|-------|------|---------|---------|
| `url` | String | required | HTTPS URL with `%searchPattern` |
| `jsonResultsKey` | String | `"streams"` | JSON field for stream array |
| `itemType` | String | required | `"info_hash"` or `"http_url"` |
| `httpUrlTargetKey` | String | — | Field for direct URL (if itemType=http_url) |
| `sizeAttr` | String | — | Dot-path to video size (e.g. `behaviorHints.videoSize`) |
| `sizePattern` | String | — | Regex to extract size from string field |
| `torrentNameTargetKey` | String | — | Dot-path to torrent/filename |
| `debridProvider` | String | — | Specific debrid to route to (null = all) |

## DebridStream Quality Regex (C1125i exact patterns)

All patterns match inside `[\s\-\.\(\[\]+...[\s\-\.\)\]]+` delimiters:

```regex
4K/2160p:    (4k|2160p)
1080p:       (1080p)
720p:        (720p)
REMUX:       (remux|bdremux)
HEVC:        (hevc|h265|x265|h\.265|x\.265|h 265|x 265)
Atmos:       (atmos)
TrueHD:      (truehd)
HDR:         (hdr|hdr10|hdr10plus|hdr10\+)
AI Upscale:  (ai[\s\-_.]*upscale(?:d)?|...vision)[\s\-_.]*ai)
DV Profile 5: (dvp5|dv\.p5|d\.v\.p\.5|dovi\.p5|...)
DV Profile 7: (dvp7|dv\.p7|d\.v\.p\.7|dovi\.p7|...)
DV:          (dv|d\.v|dovi|do\.vi|do-vi|dolby vision)
DTS-HD:      (dtshd|dts-hd|dts_hd|dts\.hd)
HDR10:       (hdr10|hdr-10|hdr_10|hdr\.10)
AV1:         (av1)
3D:          (half-ou|h-ou|sbs|hsbs)
CAM:         (telesync|telecine|tsrip|ts|hdts|...|dvdscr)
Sample:      (sample)
```

## Key Design Decisions

1. **supervisorScope isolation** — one failing addon never blocks others
2. **8s per-addon timeout** — slow addons don't delay fast ones
3. **No Stremio manifest loading** — calls stream endpoints directly (avoids manifest fetch latency)
4. **Own debrid clients** — RealDebridClient, TorBoxClient, PremiumizeClient already existed
5. **Quality at scrape-time** — not post-hoc from separate parsing pass

## To Add New Scrapers

Add an entry to `DefaultAddonScrapers.ALL`:

```kotlin
ScraperAddonConfig(
    id = "my-addon",
    name = "My Addon",
    url = "https://example.com/stream/%searchPattern.json",
    jsonResultsKey = "streams",
    itemType = "info_hash",
    sizeAttr = "behaviorHints.videoSize",
    torrentNameTargetKey = "title",
    priority = 6,
)
```

If the addon returns direct URLs instead of infoHashes, set `itemType = "http_url"` and `httpUrlTargetKey = "url"`.