# DebridStream-Style Config-Driven Scraper Pipeline (SUPERSEDED)

> **⚠️ SUPERSEDED 2026-06-01 by `debrid-agnostic-scraper-architecture.md`.**
> The in-app scraper pipeline described below has been replaced by a self-hosted
> Node.js scraper (`services/nexstream-scraper/`) that aggregates torrent sources
> and returns infoHash+magnet only. All debrid resolution now happens client-side
> via `DebridService` + `DebridEnricher` + `PlaybackOrchestrator`.
> This document is retained for historical reference only.

*Added 2026-05-31 — copied from DebridStream APK decompilation*

## Why This Exists

Torrentio's public instance (`torrentio.strem.fun`) is unreliable — 598 errors, DNS failures, format changes. The old `TorrentioProvider` embedded debrid tokens in the URL and depended on Torrentio for both scraping AND debrid resolution. When Torrentio was down, streaming was completely broken.

**Fix:** Call addon stream endpoints directly for infoHashes (not playable URLs), then resolve through our own debrid clients (RealDebridClient, TorBoxClient, etc.). This decouples us from Torrentio's availability and supports multiple addon fallbacks.

## Architecture (mirrors DebridStream)

DebridStream's scraping engine lives in `N2.q` (main orchestrator), `U2.c` (addon config), and `U2.f` (response parser):

```
ScraperAddonConfig[]     ← DebridStream U2.c — URL template + parse keys
       │
 StreamScraperEngine     ← DebridStream N2.q — parallel addon calls
  (supervisorScope)
       │
       ├─ AIOStreams  → GET .../stream/movie/tt1234567.json
       ├─ Comet       → GET .../stream/movie/tt1234567.json
       ├─ MediaFusion → GET .../stream/movie/tt1234567.json
       ├─ Jackettio   → GET .../stream/movie/tt1234567.json
       ├─ Torrentio   → GET .../stream/movie/tt1234567.json (no tokens)
       └─ PeerFlix    → GET .../stream/movie/tt1234567.json
       │
 ScraperResponseParser   ← DebridStream U2.f — config-driven JSON field extraction
  (config.jsonResultsKey, .itemType, .httpUrlTargetKey, .sizeAttr)
       │
       → List<ScraperLink>(infoHash, directUrl, title, videoSize)
       │
 DebridRouter            ← DebridStream N2.q dispatcher — route to provider SDKs
   ├─ infoHash  → checkCachePerProvider → resolveStream via first cached
   ├─ directUrl → resolveStream via first available provider
   └─ plain HTTP → return directly
       │
       → List<Stream> (deduplicated by URL/infoHash)
```

## Config-Driven Parsing

Each addon config defines how to parse its response. `ScraperResponseParser` supports dot-notation field paths like `"behaviorHints.videoSize"` — navigates `item["behaviorHints"]["videoSize"]` in the JSON tree:

| Field | Purpose | Default |
|-------|---------|---------|
| `jsonResultsKey` | Which JSON field contains stream array | `"streams"` |
| `itemType` | `info_hash` or `http_url` — extraction strategy | `"info_hash"` |
| `httpUrlTargetKey` | For `http_url` items, which field has the direct URL | `"url"` |
| `sizeAttr` | Dot-path to video size field | null |
| `torrentNameTargetKey` | Dot-path to release/display name | null |

## Key Files

| File | Purpose | LOC |
|------|---------|-----|
| `data/scraper/ScraperAddonConfig.kt` | Config model per addon (like U2.c) | 28 |
| `data/scraper/ScraperLink.kt` | Parsed link result model | 45 |
| `data/scraper/ScraperResponseParser.kt` | Config-driven JSON parser (like U2.f) | 160 |
| `data/scraper/DefaultAddonScrapers.kt` | Built-in list of 6 addon scrapers | 85 |
| `data/scraper/StreamScraperEngine.kt` | Main orchestrator (like N2.q) | 140 |
| `data/scraper/DebridRouter.kt` | Routes links through debrid resolvers | 186 |
| `data/scraper/ScraperProvider.kt` | StreamProvider wrapping the engine | 75 |

## Default Addon Scrapers

| Addon | URL Template | Priority |
|-------|-------------|----------|
| AIOStreams | `https://aiostreams.elfhosted.com/stream/%searchPattern.json` | 1 |
| Comet | `https://comet.elfhosted.com/stream/%searchPattern.json` | 2 |
| MediaFusion | `https://mediafusion.elfhosted.com/stream/%searchPattern.json` | 3 |
| Jackettio | `https://jackettio.elfhosted.com/stream/%searchPattern.json` | 4 |
| Torrentio | `https://torrentio.strem.fun/stream/%searchPattern.json` | 5 |
| PeerFlix | `https://peerflix.elfhosted.com/stream/%searchPattern.json` | 10 |

## Adding a New Scraper

```kotlin
ScraperAddonConfig(
    id = "custom-addon",
    name = "Custom Addon",
    url = "https://example.com/stream/%searchPattern.json",
    jsonResultsKey = "streams",       // which field has the array
    itemType = "info_hash",           // or "http_url"
    httpUrlTargetKey = "url",         // for http_url items
    sizeAttr = "behaviorHints.videoSize",  // dot-path to file size
    torrentNameTargetKey = "title",   // dot-path to display name
    priority = 6,
)
```

Add it to the list in `DefaultAddonScrapers.kt` and it's live.

## Debrid Router Behavior

- **infoHash + cached** → return resolved URL from first cached provider
- **infoHash + not cached** → try resolution on all providers (may add magnet)
- **directUrl + debrid domain** → unrestrict via first available provider
- **directUrl + plain HTTP** → return as-is (no debrid needed)

## Why This Beats Torrentio-Only

| Aspect | Old (TorrentioProvider) | New (ScraperProvider) |
|--------|------------------------|----------------------|
| **Dependency** | Torrentio must be up | 6 addon fallbacks |
| **Debrid** | Tokens in URL (Torrentio handles) | Our own RealDebrid/TorBox/Premiumize clients |
| **Failure mode** | 0 streams if Torrentio down | Degraded but still works |
| **Extensibility** | New addon = new Provider class | New addon = one config entry |
| **Remote update** | Requires app update | Config can be loaded from remote |
| **Response parsing** | Fixed Stremio schema | Config-driven (any JSON shape) |