---
name: stremio-scraper-pipeline
title: Stremio Scraper Pipeline
description: Build and optimize self-hosted Stremio-compatible torrent scraper addons in Node.js
domain: software-development/stremio
metadata:
  hermes:
    requires_toolsets: [terminal, web]
---

# Stremio Scraper Pipeline

How to build, optimize, and deploy a self-hosted Stremio-compatible torrent scraper addon that aggregates multiple upstream sources, deduplicates, filters, ranks, and serves results.

## Architecture Patterns

### Synchronous Pipeline (default, backward-compatible)
```
Request → CACHE CHECK → PARALLEL SCRAPE → NORMALIZE → DEDUP → DEBRID ENRICH → FILTER → RTN RANK → SORT → CAP → FORMAT → RETURN
```

### Progressive Early-Return Pipeline (preferred, faster UX)
```
Request → CACHE CHECK → FIRE ALL SCRAPERS → await first + adaptive grace → PROCESS BATCH → RETURN (partial) → [background: await rest → CACHE (full)] → [background: PREWARM NEXT]
```
- First request returns in ~200-500ms (vs 400-1500ms for full sync)
- Background populates full cache; second request hits 0ms cache
- Controlled by `?progressive=true` (default) or `?progressive=false` (sync)
- Implemented as `orchestrator.executeProgressive()` — uses `processStreamBatch()` internally for reusable batch processing

#### Adaptive Grace Period
Grace period is NOT fixed — it's calculated from health monitor data:
- Fastest scraper's EMA response time × 1.3
- Clamped between `performance.minGraceMs` (default 250) and `performance.maxGraceMs` (default 2000)
- If no health data yet (first request), falls back to `performance.progressiveGraceMs` (default 500)
- Saves ~250ms per cold first request vs fixed 500ms

```js
function calcAdaptiveGrace() {
  const healthData = getHealthMonitor().getAll();
  const fastestEma = Math.min(
    ...Object.values(healthData)
      .filter(s => s.status === 'healthy' && s.avgResponseTime > 0)
      .map(s => s.avgResponseTime)
  );
  if (fastestEma === Infinity) return 500;
  return clamp(fastestEma * 1.3, minGraceMs, maxGraceMs);
}
```

#### Health-Aware Early Return
`executeProgressive()` calls `healthMonitor.isAvailable(name)` before firing each scraper. Down scrapers (3+ consecutive failures) are skipped entirely — they don't slow down the progressive race.

#### Pre-warm Cache
After each progressive response, a fire-and-forget background task caches the next likely content:
- **Series**: Next episode (`S1E1` → `S1E2`). If episode=1, also pre-warms episode 1 of next season.
- **Movies**: Known sequel pairs from a map (18 pairs: Dark Knight → Dark Knight Rises, Infinity War → Endgame, LOTR trilogy, Nolan universe, Matrix, Star Wars, Fury Road/Furiosa, etc.)
- Fires after a 2-second delay to avoid competing with the main response.
- Configurable via `performance.prewarmNext` (default: true).

### Fan-Out Scraping
- Use `Promise.allSettled()` for parallel scraper execution — one failing never blocks others.
- Each scraper has its own timeout via `Promise.race([scraper.scrape(...), new Promise((_, reject) => setTimeout(reject, timeout))])`.
- Normalized output schema per scraper: `{ title, infoHash, fileIdx, seeders, size, tracker, source, magnet }`.

### Template-Based Scraper Config
- JSON config template loaded at startup, overridable per-request.
- Sections: `scrapers`, `debrid`, `filtering`, `ranking`, `limits`, `performance`, `presets`, `externalAddons`, `format`.
- Env vars override JSON config values.

## Optimization Techniques

| Technique | Where | Impact |
|-----------|-------|--------|
| Progressive early-return | orchestrator.js | First request 200-500ms vs 400-1500ms (returns after first scraper + grace, fills cache in background) |
| Keep-alive HTTP agent (shared axios instance) | `http-client.js` | 3× throughput on repeated requests |
| Multi-tier cache (L1 memory + L2 SQLite) | `cache.js` | 0ms vs 350ms on repeat |
| Debrid availability cache (5min TTL per hash) | orchestrator | Eliminates redundant debrid API calls |
| Request queue (inflight dedup) | `request-queue.js` | Merges concurrent duplicate requests |
| Adaptive timeouts (3× historical EMA) | orchestrator | Fails fast for slow scrapers |
| Provider health monitor (auto-disable + retry) | `health-monitor.js` | Self-healing scraper pool |
| Per-resolution result caps | orchestrator | Balance quality tiers |
| Circuit breaker (3 consecutive failures) | `health-monitor.js` | Prevents cascading failures |

## Quality Detection

### Regex Patterns
- **Resolution**: 4320p/2160p/1080p/720p/480p via regex
- **Source**: REMUX, BluRay, WEB-DL, HDTV
- **Codec**: HEVC, AV1, x265
- **HDR**: DV (P5/P7), HDR10, HDR10+, HLG
- **Audio**: Atmos, TrueHD, DTS-HD MA
- **Trash**: CAM, TS, HC, Sample, 3D

### RTN (Release Type Naming) — Trusted Release Group Scoring

RTN adds tiered reputation bonuses for known release groups:

| Tier | Groups (sample) | Bonus |
|------|----------------|-------|
| Gold 🥇 | SPARKS, FraMeSToR, RARBG, NTb, FLUX, EVO, CRiMSON, PSA, QxR, TEPES | +30 |
| Silver 🥈 | YIFY, TGx, MKVCage, eztv, AFG, SARTRE | +15 |
| Bronze 🥉 | FGT, region-specific groups | +5 |

Additional RTN scoring:
- `isProper` → +5 (Proper > standard release)
- `isRepack` → +3 (Repack > standard release)
- `isInternal` → +2
- Recency bonus: linear decay, current year = +5, 10yr-old = +0
- Year extraction from title + language detection (18 languages)
- RTN metadata stored on stream as `stream._rtn` for display in formatters

### Ranking Weights (Full Formula)
```
Score = ResolutionWeight 
       + QualityBonuses 
       + GroupTierBonus (RTN)
       + ProperBonus/RepackBonus/InternalBonus
       + RecencyBonus
       + CachedBonus 
       + min(Seeders, 500) * 0.1 
       + SourceReliabilityBonus
```
- 4K=100, 1080p=50, 720p=25, 480p=10
- REMUX=+40, DV=+20, HDR=+15, Atmos=+10, TrueHD=+8, HEVC=+5, AV1=+10
- Gold group=+30, Silver=+15, Bronze=+5
- Proper=+5, Repack=+3, Internal=+2
- Cached=+50, CAM=-200, Sample=-999
- Torrentio source=+5, Comet/MediaFusion=+3

## Debrid Integration
- **APK/client-side target**: Server returns infoHashes only. App handles debrid cache checking with user's own API keys. Recommended default for Ray's streaming app APK direction.
- **Server-side harness**: Debrid modules in `src/debrid/` with batch cache checking. Enable by setting API keys in `.env` and `enabled: true` in config. Treat this as backend/test harness unless the user explicitly wants hosted debrid logic.
- **Per-user RD key** (multi-tenant): Appends `?rdKey=USER_KEY` to stream requests. The route handler extracts it, passes through `options.rdKey` → `enrichDebridStatus(streams, rdKey)` → `client.checkCache(hashesToCheck, rdKey)`. Each user's key is used for THEIR cache checks, not stored on the server. Implemented in:
  1. `routes/stream.js` — extract `req.query.rdKey`, pass to `execute(type, id, { ..., rdKey })`
  2. `pipeline/orchestrator.js` — `enrichDebridStatus(streams, rdKey)` accepts key, passes to debrid modules
  3. `debrid/realdebrid.js` — `checkCache(hashes, userKey)` uses `userKey || config.RD_API_KEY` for API auth

  **Important**: ALL debrid module functions that make API calls must accept the optional `userKey` parameter and pass it to `rdHeaders(userKey)`. Otherwise multi-user requests silently use the server's own key.
- **Pipeline rdKey pass-through chain**: When adding a new feature that uses RD, trace the parameter through ALL layers: route → execute() → enrichDebridStatus() → client.checkCache() → debrid module API calls. Missing any link causes silent fallback to server-configured key instead of user's key.

## External Addon Aggregation

Pull streams from ANY Stremio-compatible addon alongside built-in scrapers.

**Config** (`default-config.json`):
```json
{
  "externalAddons": {
    "enabled": true,
    "addons": [
      { "name": "Peerflix", "enabled": false, "baseUrl": "https://peerflix.strem.fun", "timeout": 8000, "priority": 60 },
      { "name": "PirateBay+", "enabled": false, "baseUrl": "https://piratesbay.strem.fun", "timeout": 8000, "priority": 70 }
    ]
  }
}
```
- Each external addon gets its own health monitor (keyed `ext:<name>`), adaptive timeout, and failure isolation.
- Normalizes arbitrary Stremio stream objects into the internal format.
- Results flow through the same dedup/filter/rank pipeline.
- External addon streams without infoHash/magnet use their URL directly (e.g., HTTP-based addons).

## Formatter Styles (11 Display Styles)

Control output via `?format=<style>` query param. Each style defines what metadata appears in `name` and `description` fields.

| Style | Name format | Description |
|-------|-------------|-------------|
| `torrentio` | `[Source] Quality [Score]` | Seeders |
| `minimal` | `Quality · Group` | — |
| `standard` | `⚡/⏳ Source Quality [Score]` | (default) |
| `detailed` | Full metadata + codec/audio/HDR | Year, codec |
| `compact` | Quality only | Full info in description |
| `netflix` | `4K Ultra HD · Group` | Cinematic |
| `debrid` | `⚡CACHED/⏳UNCACHED Quality` | — |
| `info` | Source + quality + group + score | Size + seeders |
| `groups` | `🥇GroupName Quality` | Tier badge |
| `simple` | Quality only | — |
| `noEmoji` | Source Quality [Score] | No emoji |

Configurable via `format.styles.<name>` in `default-config.json`. Each style has boolean flags: `showEmoji`, `showScore`, `showSource`, `showQuality`, `showSize`, `showGroup`, `showCached`, `showSeeders`, `showTier`, `showLanguages`, `showYear`, `showCodec`.

## Optimization Techniques
Add `?preset=<name>` to stream URLs. Available presets in `default-config.json`:
- `torrentio-replacement` — Torrentio-only behavior
- `4k-enthusiast` — 4K priority, min 1080p
- `speed-demon` — 5s timeout, 20 max results
- `quality-over-quantity` — Best sources only
- `maximum-coverage` — All scrapers, full results
- `mobile` — No 4K, compact
- `anime` — Anime-optimized

## Speed Tiers
| `?speed=` | Scrapers | Timeout | Max | Use case |
|-----------|----------|---------|-----|----------|
| `fast` | Torrentio only | 5s | 30 | Quick preview |
| `balanced` | All enabled | 10s | 200 | Default |
| `comprehensive` | All + full | 15s | 500 | Power search |

## Metrics Endpoint

Two routes expose runtime metrics for monitoring and tuning:

**`GET /metrics`** — Prometheus text format (scrapeable by Grafana/Prometheus):
- `nexstream_uptime_seconds`
- `nexstream_requests_total`
- `nexstream_request_duration_ms{p50,p95,p99}`
- `nexstream_cache_l1_entries`, `nexstream_cache_l2_entries`, `nexstream_cache_l2_size_bytes`
- `nexstream_cache_hit_rate`
- Per-scraper: `nexstream_scraper_info{scraper="..."}` (status, requests, success rate, avg response, consecutive failures)

**`GET /metrics.json`** — JSON format with richer structure:
```json
{
  "server": { "version": "3.0.0", "totalRequests": 42 },
  "latency": { "p50": 0, "p95": 150, "p99": 400, "samples": 42 },
  "cache": { "l1": {...}, "l2": {...}, "hitRate": 62.5 },
  "scrapers": [{ "name": "torrentio", "status": "healthy", "avgResponseTime": 115 }]
}
```

Request times are tracked via `recordRequestTime(elapsed)` called from the stream route handler after each response (even cache hits). Rolling window of 1000 samples for percentile calculation.

## Docker Deployment
- Multi-stage build: `node:20-alpine` builder → production stage
- **Healthcheck:** Use `127.0.0.1` NOT `localhost` — inside Docker, `localhost` resolves to `::1` (IPv6) first, and apps binding to `0.0.0.0` don't listen on IPv6. Example: `wget --spider http://127.0.0.1:3091/health`
- Volume: `/app/src/data` for persistent SQLite cache
- Env vars for API keys at runtime (`docker run -e RD_API_KEY=...`)
- `--restart unless-stopped`

## Research Sources
When looking for features to adopt, check:
- `~/Unspooled-archive/` — saved competitor reference repos
- GitHub: YARR! Stremio, Comet, MediaFusion, AIOStreams, Torrentio
- Community: Tamtaro SEL templates, AIOStreams Discord, r/StremioAddons
- **Extractor catalog**: `references/scraper-engine-extractor-catalog.md` — 82 Kotlin video hoster extractors with porting recommendations
- **APK source**: `/home/rurouni/apk-analysis/scraper-engine/extractors/` — original Kotlin extractors (82 files)
- **TPB API is the only working free source** (June 2026): Torrentio, Comet, and MediaFusion are all dead. Use `https://apibay.org/q.php?q={title}&cat=0` for free info hashes. **BitSearch** (`https://bitsearch.to/search?q=...`) as secondary.
- **Standalone Ultimate Scraper** (`~/ultimate-scraper/`, port 3091): production scraper architecture shifts over time; audit the current tree before relying on old notes. Known July 2026 modular shape: 10 scrapers (TPB, Nyaa.si, AniDex, SubsPlease, AnimeTosho, YTS, 1337x, BitSearch, TokyoTosho, Torznab), multi-debrid modules (Real-Debrid, AllDebrid, Premiumize, TorBox), Stremio-compatible stream endpoint, SQLite cache in newer modular builds. Verify whether `.env` exists, TMDB/debrid keys are configured, `/api/resolve` exists, versions match, and RD cache checks are reliable. See `references/ultimate-scraper-reaudit-checklist.md` for the re-audit checklist and `references/multi-debrid-resolve-patterns.md` for provider API flows.
  - **APK target warning**: Ray clarified Ultimate Scraper is intended for use inside a streaming app APK, not only as a server-side service. Treat Express/Node routes as a test harness or optional LAN helper unless asked for backend deployment. For migration guidance, see `references/ultimate-scraper-apk-transition.md`.
  - **Simplified from legacy nexstream**: The old RTN engine, 11 formatter styles, progressive pipeline, external addon aggregation (Torrentio/Comet/MediaFusion), health monitor, speed tiers, and presets may be legacy depending on the current repo. Re-run a repo map and read current files before asserting which architecture is active.
- **Anime sources (5)**: Nyaa.si (cheerio parse — real titles/seeders/sizes), AniDex (JSON API), SubsPlease (JSON API), AnimeTosho (RSS HTML), TokyoTosho (cheerio HTML). All work server-side (no Cloudflare).
- **YTS API**: `https://yts.mx/api/v2/list_movies.json?query_term=...` — JSON API, frequently blocked server-side.
- **1337x**: Tries `torrent-api-py.onrender.com` proxy first, then falls back to direct HTML scrape of `1337x.to`. Multi-endpoint failover.
- **Free scraper (embed-based)**: `/home/rurouni/Scraper Suite/free-scraper/` — Node.js server with 6 ported extractors + Playwright headless + RC4 decryption, no debrid needed. Port 3092. Most extractors fail server-side against Cloudflare. Requires nodriver/FlareSolverr for reliable extraction.
- **Kodi addon architecture**: `references/kodi-addon-architecture-reference.md` — audited Fen Light+, Seren, Umbrella: debrid integration, scraper pipelines, caching, catalog backends (Pipelines section below has details)

## Go-Based Alternative (Non-Stremio)

For a full-stack Go scraper server (not Stremio-compatible), see `references/go-scraper-server.md`. This covers:
- Go Fiber server with TMDB catalog + 83 video hoster extractors
- Cloudflare bypass via nodriver (Python headless Chrome) — best 2026 option
- Porting Android/Kotlin scrapers to Go (TLS, crypto, regex differences)
- Streaming site frontend with iframe embed + hls.js playback
- Site-specific ID flow — the #1 mistake when porting Android scrapers. Native Android scrapers use site-internal slugs from their own search. Ported Go providers must also search the site first to get the correct slug — passing TMDB IDs directly never works.

## Web Movie Player (Python/Flask)

For a lightweight web movie streaming frontend with built-in TMDB search, The Pirate Bay API lookup, and TorrServer P2P playback, see `references/web-movie-player-flask.md`. This is useful as a Kodi-free testing interface or a simple mobile-friendly player.

### Architecture

```
Flask app (port 8800)
 ├─ TMDB API → search + movie details (metadata)
 ├─ TPB API (apibay.org) → info hashes (torrent sources)
 ├─ Jackett (opt) → Torznab indexers
 ├─ TorrServer (port 8090) → magnet → HTTP stream
 └─ Embed sources (VidSrc, 2Embed, EmbedSu, MoviesAPI, VidBinge) → instant iframe fallback
```

### Torrent Search Sources (no setup required)

**The Pirate Bay API** (free, no account, returns info hashes directly):
```
GET https://apibay.org/q.php?q={title}&cat=0
```
Returns JSON: `[{name, info_hash, seeders, size, ...}]`. The `info_hash` field is the 40-char hex BTIH. No registration needed. API is rate-limited but generous.

**Jackett** (requires self-hosted instance):
```
GET {host}:{port}/api/v2.0/indexers/all/results/torznab/api?apikey={key}&t=movie&q={title}
```
Returns Torznab XML with `<torznab:attr name="infohash">` elements. Requires installing Jackett binary and manually adding indexers via web UI at `/UI/Dashboard`.

### TorrServer Streaming Pipeline

### TPB API — No-Setup Torrent Source (Primary)

The Pirate Bay's JSON API (`apibay.org`) returns info hashes directly — no account, no configuration, free — and has replaced DMM/Zilean as the best zero-setup source:

## SanuFlix React Frontend (Alternative to Flask Templates)

For a production-quality web frontend that integrates with the Flask API backend, use the SanuFlix open-source React/Vite app (`github.com/Sanuu7/SanuFlix-opensource`):

1. **Clone** → `npm install` → configure `.env` with `VITE_TMDB_API_KEY`
2. **Configure** `src/config/servers.ts` with actual embed URLs (VidSrc, 2Embed, EmbedSu, etc.)
3. **Build** → `npm run build` → outputs to `dist/`
4. **Serve** via Flask: `send_from_directory('path/to/dist', 'index.html')`

The SanuFlix app already has: movie browsing, search, detail pages, genre filtering, server selector dropdown, Trakt-like UI, dark theme. Your Flask backend just needs to serve the static build and provide `/api/*` endpoints for TMDB search, torrent lookup, and TorrServer streaming proxy.

### Server Selector Config

The `STREAMING_SERVERS` array in `src/config/servers.ts` controls which embed sources appear in the UI dropdown. Each entry has `movieUrlTemplate` with `{tmdbId}` placeholder. The built-in server selector UI handles switching between sources seamlessly.

```
GET https://apibay.org/q.php?q={title}&cat=0
```

Response is a JSON array with fields: `name`, `info_hash`, `seeders`, `leechers`, `size`. Filter out entries where `seeders == 0` or `info_hash == '0000000000000000000000000000000000000000'`.

**Rate limiting**: Generous, no observed throttling at 1-2 req/s. TPB API has replaced the decommissioned DMM/Zilean public endpoint as the best zero-setup torrent hash source.

**Pitfall — default file_id is wrong**: Many torrents have non-video files first (subs, nfo, posters). The video is rarely file_id=0 or 1. **Always auto-detect** (see auto-detect pattern below).

**TorrServer binary** (MatriX fork, v141+):
- Download from GitHub releases, run standalone (no Docker needed if Docker has issues)
- Default port 8090, listens on 0.0.0.0
- **API endpoints** (MatriX fork specific):
  - `GET /echo` — health check, returns version string
  - `POST /torrents` — manage torrents: `{"action":"add","link":"magnet:...","save_to_db":true}`
  - `POST /torrents` — get/list: `{"action":"list"}` or `{"action":"get","hash":"..."}`
  - `GET /play/{hash}/{file_id}` — stream with auto-preloading (path format!)
  - `GET /stream/{hash}/{file_id}` — stream without preloading (path format!)
- **File stats**: response includes `file_stats` array with `{id, path, length}` for each file
- **Torrent states**: `Torrent added` → `Torrent getting info` (connecting to peers) → `Torrent working` (ready to stream)
- Peer discovery takes 10-30s for well-seeded torrents. Dead torrents never leave "getting info"

**⚠️ NOT `/api/torrents`**: The documented `/api/torrents` path returns 404 on the MatriX fork. Use bare `/torrents`.

**⚠️ NOT query params for streaming**: `/play?hash=X&file_id=Y` returns 404. Must use `/play/{hash}/{file_id}` path format.

**⚠️ Default file_id=0 or 1 is often wrong**: Many torrents have non-video files first (subs, nfo, posters). The video file is usually not file_id=0 or 1. Auto-detect by checking `file_stats` for the largest file with a video extension (`.mp4`, `.mkv`, `.avi`, `.mov`, `.webm`, `.m4v`).

**⚠️ TorrServer first-connect latency**: Peer discovery for a new magnet takes 15-30s even on well-seeded torrents. JS polling should show live status via the `/torrents` status endpoint. Torrents with 0 seeders hang in "Torrent getting info" forever — always filter `seeders > 0` before adding.

### Auto-Detect Video File

When a torrent has multiple files (subs, nfo, poster, video), default file_id=0 or 1 may point to a non-video file. Auto-detect pattern:

```python
video_exts = ('.mp4', '.mkv', '.avi', '.mov', '.webm', '.m4v')
file_stats = torrent_data.get('file_stats', [])
video_files = [f for f in file_stats if any(f.get('path','').lower().endswith(e) for e in video_exts)]
if video_files:
    video_files.sort(key=lambda f: f.get('length', 0), reverse=True)
    file_id = str(video_files[0]['id'])  # largest video file
```

### Flask Proxy for CORS-Free Streaming

Browser `<video>` elements need `Accept-Ranges: bytes` and proper Content-Type. Direct TorrServer URLs may cause CORS issues in browsers. Flask proxy pattern:

```python
@app.route('/torrent/play')
def torrent_play():
    hash_val = request.args.get('hash', '')
    # auto-detect file_id from file_stats (see above)
    ts_url = f'http://{TS_HOST}:{TS_PORT}/play/{hash_val}/{file_id}'
    ts_resp = urllib.request.urlopen(ts_url, timeout=30)
    return Response(generate(), 200, headers={
        'Content-Type': 'video/mp4', 'Accept-Ranges': 'bytes',
        'Access-Control-Allow-Origin': '*',
    })
```

### JavaScript Player Polling

TorrServer needs time to connect to peers. The player should poll:

```javascript
// Poll torrent_status API for state changes
setInterval(function() {
    fetch('/api/torrent_status?hash=' + hash).then(r => r.json()).then(d => {
        if (d.stat_string === 'Working' && d.files > 0) {
            // Torrent ready, start playing
            video.innerHTML = '<source src="/torrent/play?hash=' + hash + '">';
            video.play();
            clearInterval(pollInterval);
        }
    });
}, 1000);

// Also try HEAD request as fallback
fetch('/torrent/play?hash=' + hash, {method: 'HEAD'}).then(r => {
    if (r.ok || r.status === 206) { /* stream ready */ }
});
```

### Embed Sources (always work, instant)

| Source | URL Pattern | Reliability |
|--------|-------------|-------------|
| VidSrc | `https://vidsrc.to/embed/movie/{imdb_id}` | ★★★★★ |
| 2Embed | `https://www.2embed.cc/embed/{imdb_id}` | ★★★★☆ |
| EmbedSu | `https://embed.su/embed/movie/{imdb_id}` | ★★★★☆ |
| MoviesAPI | `https://moviesapi.club/movie/{imdb_id}` | ★★★☆☆ |
| VidBinge | `https://vidbinge.dev/embed/movie/{imdb_id}` | ★★★☆☆ |

These are third-party streaming sites with ads. For mobile, embed works well in an iframe. For desktop, consider ad-blocking browser extensions.

### Key Differences from Go-Based Scraper Server

- **Python/Flask** — zero infrastructure, no compilation needed
- **TPB API** for info hashes (no Jackett/Prowlarr required)
- **TorrServer** for P2P streaming (not hoster scrapers)
- **Multiple embed fallbacks** — always work for popular titles
- **Mobile-optimized** Tailwind CSS, dark theme, responsive
- Blueprint architecture, deployable on any machine with Python 3

## Reverse-Engineering Existing Addons

When auditing Kodi addons for architecture patterns, API keys, and debrid dependencies:

### Finding Embedded API Keys

Grep patterns for any Kodi Python addon:
```
grep -rn "client_id\s*=\s*['\"][A-Za-z0-9]\{8,\}" --include='*.py'          # OAuth client IDs
grep -rn "api.key\|apikey\|api_key\s*=" --include='*.py' --include='*.xml'  # API keys
grep -rn "default.*key\|setting_default.*[A-Za-z0-9]\{20,\}" --include='*.py' # Defaults
grep -rn "X245A4XAIBGVM\|\d\{6,12\}['\"]" --include='*.py'                  # RD generic client
grep -rn "trakt.*client\|trakt.*secret\|tmdb.*apikey\|tvdb.*apikey" --include='*.py'
```

Key files to check (in order): `kodi_utils.py` → `globals.py` → `settings_cache.py` → `indexers/tmdb.py` → `settings.xml`

### Debrid Dependency Detection

Three commands tell you if an addon requires debrid:
1. `grep -rn "if not.*debrid\|prem_providers\|debrid_enabled\|No Debrid Services"` → hard gate
2. `grep -rn "resolvers\s*=\s*{"` → resolver list with only debrid = needs it
3. `grep -rn "torrentio\|comet\|knightcrawler"` → external scrapers return magnets → needs debrid

### Adding P2P Torrent Streaming (TorrServer)

To add P2P playback to a debrid-oriented addon without modifying scrapers:

**TorrServer API** (port 8090 default):
- `GET /echo` — health check (returns `uptime`)
- `POST /torrents` — `{"action":"add","link":"magnet:..."}` → returns `{"hash":"..."}`
- `GET /play?hash=...&file_id=N` → stream URL
- `GET /stream?hash=...&index=N&link=` → alternative stream URL

**Free torrent hash sources:**
- **DMM/Zilean**: `GET https://zilean.elfhosted.com/dmm/filtered?query=TITLE&Season=N&Episode=N` — **DECOMMISSIONED Jan 2026** (404/405). Was public, no account. Replace with self-hosted Zilean (Docker `ipromknight/zilean`) or a Jackett/Prowlarr instance.
- **Jackett**: `GET {host}:{port}/api/v2.0/indexers/all/results/torznab/api?apikey=...&t=movie&q=TITLE` — Torznab XML, parse for infohash. More reliable — 50+ indexers from a single API.
- **Prowlarr**: Same Torznab protocol as Jackett. Integrates with Arr stack for indexer sync.

**Integration pattern**: Scraper finds hashes → TorrServer `add_magnet()` → TorrServer returns stream URL → Kodi player.

### Scraper Interface Adapter

When combining scrapers from different addons, two interfaces exist:
- **Interface A** (Fen Light+): `class source → def results(self, info)` — uses window property for IPC
- **Interface B** (Umbrella): `class source` with class attrs `hasMovies`/`hasEpisodes`/`pack_capable` + `def sources(self, data, hostDict)` — returns list directly

Wrap Interface B in an adapter that calls `source.sources()` from within `def results()`. See `references/kodi-addon-architecture-reference.md` for full adapter code.

### P2P-Capable Addons (for non-debrid builds)

| Addon | Engine | Repo |
|-------|--------|------|
| **Elementum** | Built-in Go binary (libtorrent) | `github.com/elgatito/plugin.video.elementum` |
| **Jacktook** | TorrServer/Torrest/Elementum/Stremio addons | `github.com/Sam-Max/plugin.video.jacktook` |
| **TorrServer** | C++ standalone daemon | `github.com/yourok/torrserver` |

## Real-Debrid Integration

See `references/real-debrid-integration.md` for complete RD integration patterns.

## Cloudflare Bypass Options (2026)

Server-side Cloudflare bypass requires a real browser. uTLS/Go HTTP alone WILL be blocked.

| Tool | Language | Pass Rate | Setup |
|------|----------|-----------|-------|
| **nodriver** (ultrafunkamsterdam) | Python | **28/31** (0 blocked) | `pip install nodriver` — direct CDP, no WebDriver |
| FlareSolverr 3.5 | Python/Docker | Good | Docker sidecar on :8191, POST API |
| chromedp | Go | OK | Needs `--no-sandbox`, 3-8s per page |

**nodriver is the best 2026 option** — successor to undetected-chromedriver, direct Chrome DevTools Protocol, no Playwright middleware detectable.

**Implementation**: call nodriver as a subprocess from Go/Node server when HTTP requests get blocked. First call takes 3-5s (Chrome startup). Use session persistence for subsequent calls.

## Scraping Site-Specific IDs (Critical)

The #1 mistake when porting Android scrapers: use site-internal IDs, not TMDB IDs.

Native Android APKs (Streamflix, etc.) work because:
1. User searches WITHIN the site (e.g., Sflix search) → gets site-specific page URLs
2. Those URLs contain the site's own content IDs
3. IDs flow through: search → detail → servers → play

When porting to server code, replicate this flow:
```
TMDB title → search SITE for title → extract SITE's page URL/ID → get servers using site ID
```
Passing raw TMDB IDs directly to site providers returns 404s or homepage HTML.

## Common Pitfalls

- **MKV → browser playback fails**: Real-Debrid unrestricted links point to MKV files (especially REMUX/BLUriP sources). The browser's `<video>` element only supports MP4/WebM/OGG. MKV container format is not supported in any browser.
  - **Fix**: Run the stream through ffmpeg with `-c copy -movflags frag_keyframe+empty_moov -f mp4 pipe:1` to transmux MKV → fragmented MP4 in real-time. This copies video/audio bitstreams (no re-encode, near-zero CPU). Add ffmpeg to Docker: `apk add --no-cache ffmpeg`.
  - If codec is still unsupported (AV1, HEVC on Chrome), fall back to direct RD link with "Open in VLC" instructions.
  - See `references/ffmpeg-stream-proxy.md` for a production-ready Node.js stream proxy route.

- **DSU dedup too aggressive**: Don't union streams sharing common words like title names. Use resolution-aware DSU (only dedup same resolution tier).
- **Debrid keys on server**: User's debrid keys should stay in the app. Server-side checking is optional and uses separate keys.
- **Test credentials: never persist to .env or config files**: When the user gives you an API key for testing, pass it at container runtime via `-e KEY=value` (docker run) or inject via terminal for a single curl test. Do NOT write test keys to `.env`, `default-config.json`, or any file that ends up committed. The user will push back: "I didn't give you permission to hardcode my key." Test keys are test-only.
- **RD API keys with `!` chars**: Real-Debrid API keys contain `!` which bash interprets as history expansion. Always wrap in single quotes: `-e 'RD_API_KEY=...'` or use `set +H` before the command.
- **Re-auditing recommended fixes**: When the user asks to re-audit prior scraper fixes, verify each recommendation against current code and live endpoints. Use `references/ultimate-scraper-reaudit-checklist.md` for the exact Ultimate Scraper checks: `.env`/TMDB state, RD cache strategy, `/api/resolve`, 1337x fallback, BitSearch parser, version consistency, tests, and health endpoints.
- **Cache key must include format**: When implementing formatter styles, append `:fmt=<style>` to the cache key. Otherwise all format requests return the same cached response.
- **Progressive cache key collision**: Progressive mode uses TWO cache keys: `cacheKey` (full results, long TTL) and `cacheKey + ':bg'` (progressive partial results, 30s TTL). Background fill clears the `:bg` key when the full cache is set.
- **Docker volume persistence**: SQLite cache survives container restarts in Docker volumes. Clear with `docker volume rm scraper-cache` for fresh tests.
- **Cross-project research**: Analyze ALL projects in the archive before implementing. Don't stop at the first one.
- **JS `name` variable shadowing**: When destructuring `const { name } = stream` and then declaring `let name = '';` in the same function scope, you get a redeclaration error. Rename the destructured property: `const { name: streamName } = stream` and use `displayName` for new variables.
- **Shell pipes and `set +H`**: When passing credentials through shell pipes, bash history expansion (`!`) can corrupt values. Use `set +H` to disable, or use single-quoted heredocs.
- **`runSingleScraper().catch()` crash**: Calling `.catch()` directly on an async function's return can throw `Cannot read properties of undefined (reading 'catch')` if the function errors synchronously before returning a Promise. Fix: wrap in `(async () => { try { return await fn(); } catch { return []; } })()` instead — this also properly catches synchronous exceptions that `.catch()` can't handle.
- **Headless browser first-call latency**: First nodriver/chromedp subprocess call is 3-5s (Chrome startup). Use session persistence or warm-up mechanisms for production.
- **RD `checkCache` / `instantAvailability` is brittle — verify current docs first**: Do NOT state that `/torrents/instantAvailability` is absent from official docs unless you have checked `https://api.real-debrid.com/` in the current session. In July 2026 the official docs still listed `GET /torrents/instantAvailability/{hash}`. The durable failure mode is silent false negatives: code that catches all RD errors and returns `false` makes cache badges lie. Safer options: log RD API error code/message and mark status unknown, or use addMagnet → selectFiles → poll info → if status==="downloaded" it was cached → unrestrict. For provider-specific flows, see `references/multi-debrid-resolve-patterns.md`.
- **Test API keys: never persist**: When testing with a user's debrid API key, pass at container runtime via `docker compose build` + `up -d` after writing the `.env`. Never commit to the repo. User will push back if key ends up in version-controlled files.

## Key Files

- `src/pipeline/orchestrator.js` — Pipeline controller; `execute()` (sync), `executeProgressive()` (progressive), `processStreamBatch()` (reusable batch processor)
- `src/pipeline/rtn.js` — RTN engine: release group extraction (150+ groups), tier detection, proper/repack/internal, year, language
- `src/pipeline/formatter.js` — 11 display styles (torrentio, minimal, standard, detailed, compact, netflix, debrid, info, groups, simple, noEmoji)
- `src/pipeline/ranker.js` — Scoring engine with RTN group bonuses
- `src/routes/stream.js` — Route handler: speed tiers, presets, format param, progressive toggle
- `src/scrapers/external-addon.js` — Generic Stremio addon fetcher for aggregation
- `src/pipeline/health-monitor.js` — Circuit breaker + EMA adaptive timeout
- `src/http-client.js` — Keep-alive connection pooling
- `src/cache.js` — L1 in-memory (NodeCache) + L2 SQLite (better-sqlite3)
- `src/config/default-config.json` — Full config: scrapers, debrid, filtering, RTN ranking, limits, presets, externalAddons, format styles, performance
- `references/kodi-addon-architecture-reference.md` — Kodi addon arch patterns: debrid integration, scraper pipelines, interface adapters, API keys, TorrServer P2P integration, combined addon builds, **web search UI for testing without Kodi**, **Kodi mock test harness patterns**
- `references/ffmpeg-stream-proxy.md` — Production-ready Node.js stream proxy for MKV->fragmented MP4 transmuxing
- `references/free-embed-scraper.md` — Free scraper server (Node.js, 6 embed extractors, no debrid)