---
name: stremio-scraper-pipeline
title: Stremio Scraper Pipeline
description: Build and optimize self-hosted Stremio-compatible torrent scraper addons in Node.js
domain: software-development/stremio
metadata:
  hermes:
    requires_toolsets: [terminal, web]
---

# Stremio Scraper Pipeline

How to build, optimize, and deploy a self-hosted Stremio-compatible torrent scraper addon that aggregates multiple upstream sources, deduplicates, filters, ranks, and serves results.

## Architecture Patterns

### Synchronous Pipeline (default, backward-compatible)
```
Request → CACHE CHECK → PARALLEL SCRAPE → NORMALIZE → DEDUP → DEBRID ENRICH → FILTER → RTN RANK → SORT → CAP → FORMAT → RETURN
```

### Progressive Early-Return Pipeline (preferred, faster UX)
```
Request → CACHE CHECK → FIRE ALL SCRAPERS → await first + adaptive grace → PROCESS BATCH → RETURN (partial) → [background: await rest → CACHE (full)] → [background: PREWARM NEXT]
```
- First request returns in ~200-500ms (vs 400-1500ms for full sync)
- Background populates full cache; second request hits 0ms cache
- Controlled by `?progressive=true` (default) or `?progressive=false` (sync)
- Implemented as `orchestrator.executeProgressive()` — uses `processStreamBatch()` internally for reusable batch processing

#### Adaptive Grace Period
Grace period is NOT fixed — it's calculated from health monitor data:
- Fastest scraper's EMA response time × 1.3
- Clamped between `performance.minGraceMs` (default 250) and `performance.maxGraceMs` (default 2000)
- If no health data yet (first request), falls back to `performance.progressiveGraceMs` (default 500)
- Saves ~250ms per cold first request vs fixed 500ms

```js
function calcAdaptiveGrace() {
  const healthData = getHealthMonitor().getAll();
  const fastestEma = Math.min(
    ...Object.values(healthData)
      .filter(s => s.status === 'healthy' && s.avgResponseTime > 0)
      .map(s => s.avgResponseTime)
  );
  if (fastestEma === Infinity) return 500;
  return clamp(fastestEma * 1.3, minGraceMs, maxGraceMs);
}
```

#### Health-Aware Early Return
`executeProgressive()` calls `healthMonitor.isAvailable(name)` before firing each scraper. Down scrapers (3+ consecutive failures) are skipped entirely — they don't slow down the progressive race.

#### Pre-warm Cache
After each progressive response, a fire-and-forget background task caches the next likely content:
- **Series**: Next episode (`S1E1` → `S1E2`). If episode=1, also pre-warms episode 1 of next season.
- **Movies**: Known sequel pairs from a map (18 pairs: Dark Knight → Dark Knight Rises, Infinity War → Endgame, LOTR trilogy, Nolan universe, Matrix, Star Wars, Fury Road/Furiosa, etc.)
- Fires after a 2-second delay to avoid competing with the main response.
- Configurable via `performance.prewarmNext` (default: true).

### Fan-Out Scraping
- Use `Promise.allSettled()` for parallel scraper execution — one failing never blocks others.
- Each scraper has its own timeout via `Promise.race([scraper.scrape(...), new Promise((_, reject) => setTimeout(reject, timeout))])`.
- Normalized output schema per scraper: `{ title, infoHash, fileIdx, seeders, size, tracker, source, magnet }`.
- **Default `fileIdx`**: Always seed `fileIdx: 0` at the provider level in the `buildResult()` helper. Don't rely on the route handler to add it — if a stream comes from an external source or custom provider without `fileIdx`, clients may refuse playback.

### Per-Provider Timeout Pattern
Slow or dead scrapers should not hold up the entire pipeline. Set a per-provider timeout, shorter than the global request timeout:

```js
const PROVIDER_TIMEOUT = 7000; // 7s max per scraper

function timeout(ms) {
  return new Promise(resolve => setTimeout(() => resolve([]), ms));
}

const promises = providers.map(async (prov) => {
  try {
    const results = await Promise.race([
      prov.search(searchQuery, context),
      timeout(PROVIDER_TIMEOUT),
    ]);
    if (!results?.length) return [];
    return results.map(r => ({ ...r, source: r.source || prov.name }));
  } catch { return []; }
});
```

Fast scrapers (TPB: ~1s) return results. Slow/timeout scrapers return empty arrays within 7s rather than blocking the response for 15s+. Total response time drops from 15s+ to ~7-8s.

### Provider Domain Fallback Pattern
Torrent sites change domains or get blocked frequently. Use multi-domain fallback arrays instead of hardcoded single URLs:

```js
const DOMAINS = ['https://eztv.re', 'https://eztv.ag', 'https://eztvx.to'];

for (const baseUrl of DOMAINS) {
  try {
    const results = await scrapeSite(baseUrl, title);
    if (results.length > 0) return results;
  } catch (err) {
    log('Provider', `${baseUrl} => ${err.message}`);
  }
}
return [];
```

Apply to: YTS (yts.rs → yts.mx), EZTV (eztv.re → eztv.ag → eztvx.to → eztv.ch), 1337x (1337x.to → 1337x.st → x1337x.se). Always use a fresh User-Agent, Referer header, and Accept header when scraping blocked sites.

### Prowlarr vs Jackett for Torznab Indexing

When choosing between Jackett and Prowlarr as the Torznab backend, prefer **Prowlarr**:

| Aspect | Jackett | Prowlarr |
|--------|---------|----------|
| **API format** | XML (Torznab RSS) | JSON REST + XML Torznab |
| **Admin API auth** | Admin password required; all API calls redirect to `/UI/Login` when password is set. API key alone doesn't work for indexer management. | `authenticationMethod: none` by default; API key works for all endpoints. |
| **Indexer management API** | PUT `/api/v2.0/indexers/{id}` — but redirects to login | POST `/api/v1/indexer` with full JSON config — works with `?apikey=` |
| **Search API** | `/api/v2.0/indexers/all/results/torznab/api?apikey=KEY&t=search&q=TITLE` — XML response, needs cheerio XML parse | `/api/v1/search?apikey=KEY&query=TITLE&type=search` — JSON response with `title`, `seeders`, `indexer`, `downloadUrl`, `categories` |
| **FlareSolverr** | Manual config in `ServerConfig.json` | Configurable via `IndexerProxies` database table or Settings → Indexers in UI |
| **Indexer count** | 624 (67 native + 557 Cardigann) | Same Cardigann definitions |
| **Web UI** | Minimal, functional | Modern, better search/lookup |

**Jackett blocking issue**: When `AdminPassword` is set, ALL API calls (including the Torznab search endpoint) work, but the admin API (`/api/v2.0/indexers`) always issues a 302 redirect to `/UI/Login`. This makes programmatic indexer configuration impossible without a session cookie. Even with `AdminPassword: null`, the redirect persists if the config file was ever created with a password.

**Preferred approach**: Run Prowlarr and query its JSON search API directly. This avoids XML parsing entirely and returns structured results with indexer names, categories, seeders, and magnet URLs.

#### Prowlarr API Integration in a Torznab Provider

The Torznab provider (in `src/providers/torrent/torznab.js`) can be rewritten to use Prowlarr's JSON API instead of Jackett's XML response:

```js
// Prowlarr API: GET /api/v1/search?apikey=KEY&query=TITLE&type=search&limit=100
// Returns JSON array with: title, seeders, size, indexer, downloadUrl, guid, infoUrl, infoHash, categories, publishDate

const { data } = await http.get(`${baseUrl}/api/v1/search`, {
  params: { apikey: apiKey, query: title, type: 'search', limit: 100 },
  timeout: 20000,
});

for (const r of data) {
  // Extract infohash — Prowlarr returns `infoHash` field directly, plus guid/downloadUrl/infoUrl
  let hash = (r.infoHash || '').toLowerCase();
  if (!hash || !/^[a-fA-F0-9]{40}$/.test(hash)) {
    hash = (r.downloadUrl || '').match(/magnet:\?xt=urn:btih:([a-fA-F0-9]{40})/)?.[1]
      || r.guid?.match(/[a-fA-F0-9]{40}/)?.[0]
      || r.infoUrl?.match(/[a-fA-F0-9]{40}/)?.[0];
  }
  hash = (hash || '').toLowerCase();
  if (!hash || !/^[a-f0-9]{40}$/.test(hash)) continue;
  
  const seeds = r.seeders || 0;
  const size = r.size || 0;
  const source = r.indexer || 'Prowlarr';
}
```

**Benefits over Jackett XML**: No cheerio XML parsing, no `<torznab:attr>` XPath issues, no XML entity encoding problems, direct access to structured JSON fields.

#### Adding Indexers to Prowlarr Programmatically

```bash
PROWLARR_KEY="your_api_key"

# Add a Cardigann indexer
curl -s -X POST "http://localhost:9696/api/v1/indexer?apikey=$PROWLARR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "implementation": "Cardigann",
    "configContract": "CardigannSettings",
    "name": "The Pirate Bay",
    "definitionName": "thepiratebay",
    "fields": [
      {"name": "definitionFile", "value": "thepiratebay"},
      {"name": "apiurl", "value": "apibay.org"}
    ],
    "appProfileId": 1,
    "priority": 25,
    "protocol": "torrent",
    "privacy": "public",
    "enable": true,
    "supportsRss": true,
    "supportsSearch": true
  }'
```

**Critical fields for Cardigann indexers**:
- `definitionFile` must match `definitionName` (hidden required field)
- `priority` must be between 1-50 (0 causes a validation error)
- `appProfileId`: find available profiles via `GET /api/v1/appProfile`
- `fields`: include `baseUrl` for most Cardigann indexers, `apiurl` for TPB/YTS

**Indexer definitions that work** (July 2026):
- `thepiratebay` (uses apibay.org API — JSON, not web scrape)
- `limetorrents` (no Cloudflare)
- `yts` (uses yts.mx/api — JSON API)
- Most others (1337x, kickasstorrents, eztv) are Cloudflare-blocked even through Prowlarr

#### Prowlarr Search for All Enabled Indexers

```bash
# Search all configured indexers
curl -s "http://localhost:9696/api/v1/search?apikey=KEY&query=inception&type=search&limit=100"
```

The "all" aggregate Torznab indexer (`/0/api`) only returns 1 test result. Always use the JSON search API for real queries.

#### FlareSolverr Integration with Prowlarr

FlareSolverr helps bypass Cloudflare on protected indexers. Install alongside Prowlarr:

```bash
docker run -d --name flaresolverr --restart unless-stopped -p 8191:8191 \
  ghcr.io/flaresolverr/flaresolverr:latest
```

**Method 1 — SQLite database injection** (when UI is inaccessible):
```python
import sqlite3, json
conn = sqlite3.connect('/path/to/prowlarr.db')
cursor = conn.cursor()
settings = json.dumps({
    'name': 'FlareSolverr',
    'implementation': 'FlareSolverr',
    'configContract': 'FlareSolverrSettings',
    'tags': [],
    'fields': [
        {'name': 'flareSolverrUrl', 'value': 'http://127.0.0.1:8191/v1'},
        {'name': 'flareSolverrMaxTimeout', 'value': 55000}
    ]
})
cursor.execute('''INSERT OR REPLACE INTO IndexerProxies (Id, Name, Settings, Implementation, ConfigContract, Tags)
    VALUES (?, ?, ?, ?, ?, ?)''', (1, 'FlareSolverr', settings, 'FlareSolverr', 'FlareSolverrSettings', '[]'))
conn.commit()
```

**Method 2 — Web UI**: Settings → Indexers → FlareSolverr section.

**Known limitation**: Even with FlareSolverr, many Cardigann indexers (1337x, KAT, EZTV) remain blocked because their proxy domains use Cloudflare JS challenges that exceed the 55s max timeout.

#### Config Variables

Set in `.env`:
```
JACKETT_URL=http://127.0.0.1:9696
JACKETT_API_KEY=your_prowlarr_api_key
```

The variable names `JACKETT_*` are legacy but reused — Prowlarr uses the same Torznab-compatible `apikey` parameter.

### Template-Based Scraper Config
- JSON config template loaded at startup, overridable per-request.
- Sections: `scrapers`, `debrid`, `filtering`, `ranking`, `limits`, `performance`, `presets`, `externalAddons`, `format`.
- Env vars override JSON config values.

## Optimization Techniques

| Technique | Where | Impact |
|-----------|-------|--------|
| Progressive early-return | orchestrator.js | First request 200-500ms vs 400-1500ms (returns after first scraper + grace, fills cache in background) |
| Keep-alive HTTP agent (shared axios instance) | `http-client.js` | 3× throughput on repeated requests |
| Multi-tier cache (L1 memory + L2 SQLite) | `cache.js` | 0ms vs 350ms on repeat |
| Debrid availability cache (5min TTL per hash) | orchestrator | Eliminates redundant debrid API calls |
| Request queue (inflight dedup) | `request-queue.js` | Merges concurrent duplicate requests |
| Adaptive timeouts (3× historical EMA) | orchestrator | Fails fast for slow providers |
| Provider health monitor (auto-disable + retry) | `health-monitor.js` | Self-healing scraper pool |
| Per-resolution result caps | orchestrator | Balance quality tiers |
| Circuit breaker (3 consecutive failures) | `health-monitor.js` | Prevents cascading failures |
| **Pre-warm TorrServer at scrape time** | torr.routes.js | Top 3 hashes pre-added → click plays at ~0.1s instead of 3-4s cold start |
| **Wait-first TorrServer proxy** | torr.routes.js | Wait for TS ready BEFORE sending response → no empty-body gap → no 502/504 from upstream proxies |
| **Domain rotation + cooldown** | domainRotation.js | Auto-fallback on 429/403, 5min error cooldown, 10min rate-limit cooldown |
| **Per-provider rate limiting** | httpClient.js | Bottleneck: 2 req/s per provider with max 2 concurrent |
| **Early return on enough results** | pipeline/scrapePipeline.js | Returns ≥10 results by 6s without waiting for slow providers |
| **Content-based title filtering** | contentFilter.js | Drops results whose title doesn't match the requested movie/show title |
| **Codec-aware scoring for non-debrid** | metadataMatcher.js, scoring.js | AVC/H264 preferred over HEVC/AV1 — device-dependent HW decode |
| **TorrServer PreloadCache tuning** | settings.json | PreloadCache=90 with 4GB cache = 3.6GB pre-buffer → 30-60s delay. Set to 5% with 1GB cache = 50MB pre-buffer → instant 1-3s playback |

## Quality Detection

### Regex Patterns
- **Resolution**: 4320p/2160p/1080p/720p/480p via regex
- **Source**: REMUX, BluRay, WEB-DL, HDTV
- **Codec**: HEVC, AV1, x265
- **HDR**: DV (P5/P7), HDR10, HDR10+, HLG
- **Audio**: Atmos, TrueHD, DTS-HD MA
- **Trash**: CAM, TS, HC, Sample, 3D

### Metadata Matching Layer (TMDB + TVDB)

Strict title/year/season/episode matching backed by TMDB (primary) and TVDB (backup for series validation).

#### TMDB Client — Two-Auth Support
Supports v3 API key (query param) and v4 Bearer token (preferred). Methods: `fetchById()`, `fetchByTmdbId()`, `getAliases()`, `getTranslations()`, `getSeasonDetails()`, `getSeasons()`. Secrets never logged.

#### TVDB Client — Series Validation Backup (v4 API)
JWT token auth (login → Bearer, cached 12h). Methods: `getSeriesByImdb()`, `getSeasonEpisodes()`, `validateEpisode()`. Use when TMDB lacks episode data.

#### Matching Algorithm (`metadataMatcher.js`)
- **Title normalization**: `cleanTitle()` strips `.`/`_`/`-` separators and trailing `-GroupName` before comparison
- **Similarity**: Jaccard on core words (stop words removed). Thresholds: movie ≥0.85, series ≥0.65
- **Year matching**: ±1 accept, ±2 with 0.7× penalty, >2 reject
- **Remake/reboot**: All context words match + year diff >3 = reject as wrong entry
- **Sequel detection**: Extra words matching sequel indicators (ii, iii, 2, reloaded, revolutions, returns, chapter) = reject
- **Season packs**: Accepted matching requested season (score 0.8-0.9)
- **Episode-specific** (`tt:1:2`): Exact S/E match required; season pack accepted with note
- **Conditional filter**: Remove bad/unknown-quality/no-seeder results only when ≥5 good exist — prevents empty results for obscure content
- **Stream scoring**: `scoreResults()` computes `streamScore` per: cached(+25, +5 multi) + resolution(4K=100,1080p=75,720p=55) + quality(REMUX=100,BluRay=90,WEB-DL=80) + encode(H264=100,AVC=100,HEVC=50,AV1=30) + HDR(DV=15,HDR10+=12,HDR10=10,HDR=8) + audio(Atmos=5,TrueHD=4,DTS-HD MA=4,DTS:X=4,DD+=2) + matchScore*15

#### Enhanced Release Parser (`releaseParser.js`)
Extracts from torrent titles: resolution, quality source (REMUX/BluRay/WEB-DL/HDTV), codec (HEVC/AV1/H264/XviD/VP9), HDR in priority order (DV > HDR10+ > HDR10 > HDR > HLG), DV profile (P5/P7/P8 via `DOVI.*Profile [578]`), audio (Atmos/TrueHD/DTS-HD MA/DTS:X/FLAC/DD+/OPUS/AAC), channels (7.1/5.1/2.0), languages (22 tags, deduplicated), subtitles, edition (Extended/Uncut/Director's Cut/Remastered/IMAX/Anniversary/4K Scan), release group, pack detection.

### RTN (Release Type Naming) — Trusted Release Group Scoring

RTN adds tiered reputation bonuses for known release groups:

| Tier | Groups (sample) | Bonus |
|------|----------------|-------|
| Gold 🥇 | SPARKS, FraMeSToR, RARBG, NTb, FLUX, EVO, CRiMSON, PSA, QxR, TEPES | +30 |
| Silver 🥈 | YIFY, TGx, MKVCage, eztv, AFG, SARTRE | +15 |
| Bronze 🥉 | FGT, region-specific groups | +5 |

Additional RTN scoring:
- `isProper` → +5 (Proper > standard release)
- `isRepack` → +3 (Repack > standard release)
- `isInternal` → +2
- Recency bonus: linear decay, current year = +5, 10yr-old = +0
- Year extraction from title + language detection (18 languages)
- RTN metadata stored on stream as `stream._rtn` for display in formatters

### Ranking Weights (Full Formula)
```
Score = ResolutionWeight 
       + QualityBonuses 
       + GroupTierBonus (RTN)
       + ProperBonus/RepackBonus/InternalBonus
       + RecencyBonus
       + CachedBonus 
       + min(Seeders, 500) * 0.1 
       + SourceReliabilityBonus
```
- 4K=100, 1080p=50, 720p=25, 480p=10
- REMUX=+40, DV=+20, HDR=+15, Atmos=+10, TrueHD=+8, AVC/H264=+10, HEVC=+5, AV1=+0
- Gold group=+30, Silver=+15, Bronze=+5
- Proper=+5, Repack=+3, Internal=+2
- Cached=+50, CAM=-200, Sample=-999
- Torrentio source=+5, Comet/MediaFusion=+3

## Debrid Provider Research & Cross-Reference Methodology

When auditing or fixing debrid service integration, cross-reference **at least three sources** before making changes:

1. **Official API docs** of the service (check current — not a cached version)
2. **Active open-source implementations** (clone and read the actual code)
3. **The API docs again** — after reading the code, re-read to confirm

### Reference Codebases (most actively maintained, July 2026)

| Repo | Stars | Debrid Count | Notes |
|------|-------|-------------|-------|
| `github.com/TheBeastLT/torrentio-scraper` | ~300 | 8 (RD, AD, PM, TB, DL, ED, Offcloud, Putio) | **Most comprehensive**. Uses dedicated npm packages (`real-debrid-api`, `premiumize-api`, `all-debrid-api`). Community Torrentio CE — actively maintained. |
| `github.com/peterdsp/Magnetio` | 19 | 8 (same set) | Fork of torrentio-scraper with raw axios. Simpler code, easier to borrow patterns from. |
| `github.com/mhdzumair/MediaFusion` | 898 | 4 (RD, AD, PM, TB) | Python/Rust/TypeScript stack. Different architecture but same debrid API patterns. |

### Cross-Reference Workflow

```bash
# 1. Clone each repo
git clone --depth 1 https://github.com/TheBeastLT/torrentio-scraper.git /tmp/torrentio
git clone --depth 1 https://github.com/peterdsp/Magnetio.git /tmp/magnetio

# 2. Compare per-file — if both references disagree with your code, yours is wrong
diff /tmp/magnetio/addon/moch/realdebrid.js /tmp/torrentio/addon/moch/realdebrid.js | head -40
diff /tmp/magnetio/addon/moch/premiumize.js /tmp/torrentio/addon/moch/premiumize.js | head -40

# 3. Check official API docs
curl -s https://api.real-debrid.com/ | grep -i "instantAvailability\|addMagnet"
```

### Verification Techniques

- **Disagreement**: If both reference implementations use a different HTTP method/param format than your code, yours is almost certainly wrong
- **Undocumented endpoints**: AllDebrid `/v4/magnet/instant` is undocumented but used by ALL implementations (torrentio, Magnetio, Comet, Debrid Media Manager) — stable for years, but has no deprecation notice if it breaks
- **Official fallback**: AllDebrid's official cache check flow is: upload magnet → check `ready` field in upload response → poll `/v4.1/magnet/status`
- **Test the endpoint directly**: `curl` with both old and new method/params to compare responses
- **Check changelogs**: AllDebrid removed the `agent` requirement Jan 2025, Premiumize now recommends `Authorization: Bearer` over query param `apikey`

### Critical Findings from this Session (July 2026)

| Service | We Had | Reference Shows | Impact |
|---------|--------|----------------|--------|
| **Premiumize cache check** | `GET /api/cache/check` with `items[]` | **`POST`** with `items[][src]` (API docs + both refs) | ❌ Broken — both refs use POST |
| **TorBox checkcached** | `GET ?hash=single` | **`POST { hashes: [] }`** (batch array) | ❌ Wrong — POST with array is correct |
| **Real-Debrid** | `addMagnet` → `selectFiles` → poll | Same + check existing torrents list first | ✅ Correct, enhanced |
| **AllDebrid** | `GET /v4/magnet/instant?magnets[]=` | **`POST`** same endpoint (both refs) | ⚠️ GET works but POST matches refs |
| **AllDebrid auth** | Query param `apikey=` | Same + `Authorization: Bearer` (docs) | ✅ Works, legacy but accepted |
| **RD status checks** | `downloaded` or `waiting_files_selection` | Same — `magnet_error`, `magnet_conversion`, `downloaded`, `dead`, etc. | ✅ Correct |

### Per-Service Error Codes (for robust handling)

| Service | Auth Errors | Limit Errors | Content Errors |
|---------|------------|-------------|----------------|
| **Real-Debrid** | 8, 9, 20 | 21, 23, 26, 36 | 29 (too big), 35 (infringing) |
| **AllDebrid** | `AUTH_BAD_APIKEY`, `AUTH_BLOCKED`, `AUTH_USER_BANNED` | `MAGNET_TOO_MANY` | `MAGNET_INVALID_URI`, status code 8 (too big) |
| **Premiumize** | `'Not logged in.'`, `'Account not premium.'` | Fair use limit, max 25 active downloads, space full | — |
| **TorBox** | `BAD_TOKEN`, `AUTH_ERROR`, `PLAN_RESTRICTED_FEATURE` | `MONTHLY_LIMIT`, `COOLDOWN_LIMIT`, `ACTIVE_LIMIT` | `DOWNLOAD_TOO_LARGE` |

### Smart Video File Selection Pattern

When a torrent has multiple files (subs, nfo, poster, video), always auto-select the largest video file rather than relying on `fileIdx`:

```js
const VIDEO_EXTS = /\.(mp4|mkv|avi|mov|wmv|flv|webm|m4v|ts|m2ts)$/i;
const files = torrent.files || [];
const videos = files.filter(f => VIDEO_EXTS.test(f.path || f.name || ''));
const target = videos.sort((a, b) => (b.size || 0) - (a.size || 0))[0];
```

This matches the behavior of ALL reference implementations (torrentio-scraper, Magnetio).

## Debrid Integration
- **APK/client-side target**: Server returns infoHashes only. App handles debrid cache checking with user's own API keys. Recommended default for Ray's streaming app APK direction.
- **Server-side harness**: Debrid modules in `src/debrid/` with batch cache checking. Enable by setting API keys in `.env` and `enabled: true` in config. Treat this as backend/test harness unless the user explicitly wants hosted debrid logic.
- **Per-user RD key** (multi-tenant): Appends `?rdKey=USER_KEY` to stream requests. The route handler extracts it, passes through `options.rdKey` → `enrichDebridStatus(streams, rdKey)` → `client.checkCache(hashesToCheck, rdKey)`. Each user's key is used for THEIR cache checks, not stored on the server. Implemented in:
  1. `routes/stream.js` — extract `req.query.rdKey`, pass to `execute(type, id, { ..., rdKey })`
  2. `pipeline/orchestrator.js` — `enrichDebridStatus(streams, rdKey)` accepts key, passes to debrid modules
  3. `debrid/realdebrid.js` — `checkCache(hashes, userKey)` uses `userKey || config.RD_API_KEY` for API auth

  **Important**: ALL debrid module functions that make API calls must accept the optional `userKey` parameter and pass it to `rdHeaders(userKey)`. Otherwise multi-user requests silently use the server's own key.
- **Pipeline rdKey pass-through chain**: When adding a new feature that uses RD, trace the parameter through ALL layers: route → execute() → enrichDebridStatus() → client.checkCache() → debrid module API calls. Missing any link causes silent fallback to server-configured key instead of user's key.

## Stremio Catalog Integration

A scraper addon needs catalogs to appear in Stremio's Discover tab. Without catalogs, the addon only responds when the user already typed a search.

### Manifest

Declare `resources: ['stream', 'catalog']` and a `catalogs` array:

```json
{
  "resources": ["stream", "catalog"],
  "types": ["movie", "series"],
  "catalogs": [
    {
      "id": "trending-movie",
      "name": "Trending Movies",
      "type": "movie",
      "extra": [{ "name": "skip", "isRequired": false }]
    }
  ]
}
```

The `id` becomes the URL: `/catalog/{type}/{id}.json?skip={N}`.

### TMDB-Powered Route (Free)

Mount under `/` (not `/api`) so Stremio's addon resolver finds it:

```js
app.use('/', catalogRoutes);
```

**Auth — support both v3 API key (query param) and v4 Bearer token (header):**

```js
const CATALOG_MAP = {
  'trending-movie':  { endpoint: '/trending/movie/week', type: 'movie' },
  'trending-series': { endpoint: '/trending/tv/week',    type: 'series' },
  'popular-movie':   { endpoint: '/movie/popular',       type: 'movie' },
  'popular-series':  { endpoint: '/tv/popular',          type: 'series' },
  'top-movie':       { endpoint: '/movie/top_rated',     type: 'movie' },
  'top-series':      { endpoint: '/tv/top_rated',        type: 'series' },
};

router.get('/catalog/:type/:id.json', async (req, res) => {
  const catalog = CATALOG_MAP[req.params.id];
  if (!catalog) return res.json({ metas: [] });

  const page = Math.floor(parseInt(req.query.skip || '0', 10) / 20) + 1;
  const headers = {};
  let path = `${catalog.endpoint}?language=en-US&page=${page}`;

  if (TMDB_ACCESS_TOKEN) {
    headers.Authorization = `Bearer ${TMDB_ACCESS_TOKEN}`;
  } else if (TMDB_API_KEY) {
    path += `&api_key=${TMDB_API_KEY}`;
  } else {
    return res.json({ metas: [] });  // no auth = empty
  }

  const data = await fetch(`https://api.themoviedb.org/3${path}`, {
    headers, signal: AbortSignal.timeout(5000),
  }).then(r => r.ok ? r.json() : null);

  if (!data?.results) return res.json({ metas: [] });

  res.json({
    metas: data.results.map(item => ({
      id: `tmdb:${item.id}`,
      type: req.params.type,
      name: item.title || item.name,
      poster: item.poster_path ? `https://image.tmdb.org/t/p/w342${item.poster_path}` : undefined,
      background: item.backdrop_path ? `https://image.tmdb.org/t/p/w780${item.backdrop_path}` : undefined,
      posterShape: 'poster',
      description: item.overview || '',
      year: (item.release_date || item.first_air_date || '').slice(0, 4),
      imdbRating: item.vote_average ? Math.round(item.vote_average * 10) / 10 : undefined,
    })),
    hasMore: data.page < data.total_pages,
  });
});
```

**Pitfall — catalog URL mismatch**: If catalog IDs are `trending-movie`, the URL is `/catalog/movie/trending-movie.json`. Testing with `/catalog/movie/trending.json` silently returns empty metas because the `CATALOG_MAP` lookup misses.

**Pitfall — empty results = auth failure**: `{"metas":[]}` most commonly means missing TMDB auth. Test directly:
```bash
curl -s "https://api.themoviedb.org/3/trending/movie/week?api_key=YOUR_KEY" | \
  python3 -c "import json,sys; d=json.load(sys.stdin); print('Results:', len(d.get('results',[])))"
```

**Types restriction**: Stremio catalogs only support `movie` and `series`. Remove `anime` from `manifest.types` when adding catalogs, or serve empty arrays for unsupported types. Anime works for streams — just not catalogs.

**Pagination**: Stremio sends `?skip=20`, `?skip=40`. Convert: `Math.floor(skip / 20) + 1`.

## External Addon Aggregation

Pull streams from ANY Stremio-compatible addon alongside built-in scrapers.

**Config** (`default-config.json`):
```json
{
  "externalAddons": {
    "enabled": true,
    "addons": [
      { "name": "Peerflix", "enabled": false, "baseUrl": "https://peerflix.strem.fun", "timeout": 8000, "priority": 60 },
      { "name": "PirateBay+", "enabled": false, "baseUrl": "https://piratesbay.strem.fun", "timeout": 8000, "priority": 70 }
    ]
  }
}
```
- Each external addon gets its own health monitor (keyed `ext:<name>`), adaptive timeout, and failure isolation.
- Normalizes arbitrary Stremio stream objects into the internal format.
- Results flow through the same dedup/filter/rank pipeline.
- External addon streams without infoHash/magnet use their URL directly (e.g., HTTP-based addons).

## Stremio Stream Protocol — behaviorHints and Embed Format

Stremio stream objects have two display fields (`name`, `description`) and several protocol fields. Getting these right determines whether clients like Stremio, Nuvio TV, and third-party apps can render streams.

### Required Fields Per Stream Type

| Field | Torrent streams | Embed/direct streams | Notes |
|-------|----------------|---------------------|-------|
| `name` | ✅ Required | ✅ Required | Display label in stream list |
| `description` | ✅ Recommended | ⚠️ Optional | Full metadata shown on hover |
| `infoHash` | ✅ Required | ❌ N/A | 40-char hex BTIH |
| `url` | ❌ N/A | ✅ Required | Direct play URL or embed page URL |
| `fileIdx` | ✅ Recommended | ❌ N/A | File index within torrent (default: 0) |
| `behaviorHints` | ✅ Recommended | ✅ Required for embeds | Controls client behavior |
| `subtitles` | ⚠️ Optional | ⚠️ Optional | Array of subtitle objects |

### behaviorHints Field

Controls how the client displays and selects streams:

```json
{
  "behaviorHints": {
    "bingeGroup": "provider:TPB",
    "defaultVideo": true,
    "filename": "The.Matrix.1999.2160p.mkv",
    "notWebReady": false
  }
}
```

| Key | Type | Purpose |
|-----|------|---------|
| `bingeGroup` | `string` | Groups streams so only one per group shows in autoplay. Use `provider:{name}` for torrent sources, unique ID for embeds |
| `defaultVideo` | `boolean` | Mark the best stream as default (only one stream should have this) |
| `filename` | `string` | Torrent filename hint — helps identify the file when multiple torrents share a hash |
| `notWebReady` | `boolean` | **Critical for embed streams**: `true` means the URL is a web page (iframe), not a direct video. Clients open it in a WebView/iframe instead of a native player |

### Embed Stream Format (Nuvio TV / Stremio Compatibility)

**WRONG** — using custom fields that clients don't recognize:
```json
{
  "name": "[EMBED] SuperEmbed",
  "url": "https://multiembed.mov/?video_id=603&tmdb=1",
  "embedType": "iframe",     // ❌ AIOStreams custom field, not standard
  "behaviorHints": { "notWebReady": false }
}
```

**RIGHT** — standard Stremio protocol:
```json
{
  "name": "SuperEmbed",
  "url": "https://multiembed.mov/?video_id=603&tmdb=1",
  "behaviorHints": { "bingeGroup": "superembed", "notWebReady": true }
}
```

The `notWebReady: true` tells the client "I cannot play this directly in a video player — open it in a WebView/iframe." Always set this on embed/iframe streams. Do NOT use a non-standard `embedType` field — it will be ignored by clients that don't know about it.

### Embed Source URLs

| Source | URL Pattern | Nuvio TV | Notes |
|--------|-------------|----------|-------|
| SuperEmbed | `https://multiembed.mov/?video_id={tmdbId}&tmdb=1` | ⚠️ Only if seapi.link resolves to direct URL | seapi.link API often unreachable (HTTP 000) |
| VidSrc | `https://vidsrc.to/embed/{type}/{imdbId}` | ⚠️ Only if RC4 decrypt succeeds for HLS URL | Server-side RC4 decryption works ~50% of the time |
| 2Embed | `https://www.2embed.cc/embed/{imdbId}` | ❌ Always iframe — do not include | No extractor available |
| EmbedSu | `https://embed.su/embed/{type}/{imdbId}` | ❌ Always iframe — do not include | No extractor available |
| VidBinge | `https://vidbinge.dev/embed/{type}/{imdbId}` | ❌ Always iframe — do not include | No extractor available |

All are iframe embeds — always use `behaviorHints: { notWebReady: true }`.

## Formatter Styles (11 Display Styles)

Control output via `?format=<style>` query param. Each style defines what metadata appears in `name` and `description` fields.

| Style | Name format | Description |
|-------|-------------|-------------|
| `torrentio` | `[Source] Quality [Score]` | Seeders |
| `minimal` | `Quality · Group` | — |
| `standard` | `⚡/⏳ Source Quality [Score]` | (default) |
| `detailed` | Full metadata + codec/audio/HDR | Year, codec |
| `compact` | Quality only | Full info in description |
| `netflix` | `4K Ultra HD · Group` | Cinematic |
| `debrid` | `⚡CACHED/⏳UNCACHED Quality` | — |
| `info` | Source + quality + group + score | Size + seeders |
| `groups` | `🥇GroupName Quality` | Tier badge |
| `simple` | Quality only | — |
| `noEmoji` | Source Quality [Score] | No emoji |

Configurable via `format.styles.<name>` in `default-config.json`. Each style has boolean flags: `showEmoji`, `showScore`, `showSource`, `showQuality`, `showSize`, `showGroup`, `showCached`, `showSeeders`, `showTier`, `showLanguages`, `showYear`, `showCodec`.

## Stream Display Formatting

Stremio stream objects use two display fields:
- **`name`** — compact badge line shown in the stream list
- **`description`** — full metadata shown on hover/expand

### Name Field Convention

| Component | Example | Meaning |
|-----------|---------|---------|
| Resolution badge | `4K`, `1080P`, `720P` | Video resolution (left-padded for alignment) |
| Type marker | `⁽ᵖ²ᵖ⁾`, `⁽ʷᵉᵇ⁾` | Stream type |
| Cache indicator | `⚡` / `⏳` | Cached on debrid vs not |
| Quality tag | `〈Remux〉`, `〈BluRay〉` | Release quality |
| Private marker | `🔒` | Private stream |

**Pattern**: `{resolution_badge} {type_marker} {cache_badge} {quality_tag}`

### Description Field Structure

Multi-line, one metadata category per line:

```
✎ The Matrix (1999) S1 E1
▣ HEVC • ✦ DV · HDR10+
♬ TrueHD · Atmos · 7.1
35.35 GB • ⇄ 85❦
⛿ English · Japanese (English subs)
[TPB] · GalaxyRG265 [109.5]
```

| Line | Content |
|------|---------|
| 1 | Title, year, season/episode |
| 2 | Codec + HDR/visual tags + edition |
| 3 | Audio codecs + channels |
| 4 | Size + bitrate + seeders + age |
| 5 | Languages + subtitles |
| 6 | Source + release group + score |

### Implementation Pattern

```js
function buildStreamName(stream, parsed, cached) {
  const parts = [];
  parts.push(formatResolution(stream.quality));
  parts.push(formatTypeMarker(stream.type));
  parts.push(cached ? '⚡' : '⏳');
  parts.push(formatQualityTag(stream.quality));
  return parts.join(' ').trim();
}
function buildStreamDescription(stream, parsed, cached, ctx) {
  const lines = [];
  lines.push(`✎ ${ctx.title} (${ctx.year})`);
  if (parsed.codec || parsed.isDV || parsed.isHDR)
    lines.push(`▣ ${parsed.codec} • ✦ ${hdrTags(parsed)}`);
  if (parsed.audio?.length)
    lines.push(`♬ ${parsed.audio.join(' · ')}`);
  lines.push(`${formatSize(stream.size)} • ⇄ ${stream.seeders}❦`);
  lines.push(`[${stream.source}] · ${parsed.releaseGroup||''} [${stream.finalScore}]`);
  return lines.filter(Boolean).join('\n');
}
```

See `references/stremio-stream-formatter-patterns.md` for full implementation with resolution badges, cache indicators, HDR type detection, audio channel extraction, quality tag mapping, and the complete formatter code.

## Smart Deduplication

Three-key dedup strategy matching AIOStreams' approach:

### Key 1: infoHash (exact match)
```js
const ih = r.infoHash.toLowerCase();
if (seenInfoHashes.has(ih)) continue;
```

### Key 2: Filename (normalized)
```js
const fn = r.title.toLowerCase().replace(/[^a-z0-9]/g, '').trim();
if (seenFilenames.has(fn)) continue;
```

### Key 3: Smart Fingerprint (10-attribute composite)
Builds a composite key from normalized attributes. Two streams with the same fingerprint are the same release from different sources:

```
sz:{size/100MB rounded}|res:{quality}|ql:{source}|cd:{codec}|vt:{DV,HDR,IMAX}|at:{Atmos,TrueHD}|ac:{7.1}
```

**Attributes**: size (±10% tolerance via rounding to nearest 100MB), resolution, quality source, codec, visualTags (DV/HDR/IMAX), audioTags, audioChannels.

**Behavior**: When two streams share a fingerprint, keep the one with higher (seeders + score). Optionally dedup across sources (`multiGroupBehaviour: 'aggressive'`).

**When to apply**: After all scrapers return, before ranking. Apply across both internal scrapers AND external addon results. Apply per-resolution tier to avoid downscaling (keep both 4K and 1080p even if same movie).

## Scoring and Ranking — Split Sort

Rather than a single score formula, use separate sort chains for cached vs uncached:

| Tier | Primary sort | Secondary | Tertiary |
|------|-------------|-----------|----------|
| **Cached** | score (quality + HDR + audio + codec) | seeders | age (asc) |
| **Uncached** | score (quality-driven) | seeders | age (asc) |

This ensures 4K REMUX appears before high-seeded 720p, even when uncached.

**Implementation pattern**: split the ranked results array, sort each half independently, then concat cached first.

## Size and Bitrate Limits per Resolution

Apply per-resolution caps to filter suspicious results:

| Resolution | Min size | Max size |
|-----------|----------|----------|
| 4K / 2160p | 500 MB | 150 GB |
| 1440p | 300 MB | 100 GB |
| 1080p | 100 MB | 60 GB |
| 720p | 50 MB | 20 GB |
| 480p | 30 MB | 5 GB |
| SD | 10 MB | 2 GB |

Filter out results outside these bounds before ranking.

## Per-Resolution Result Capping

Without caps, a popular movie can return 40+ streams. Cap per resolution to keep the UI manageable:

```js
const caps = {
  '4K': 8, '2160p': 8, '1440p': 4,
  '1080p': 12, '720p': 6, '480p': 4, 'SD': 3,
};

function capPerResolution(results) {
  const counts = {};
  const out = [];
  for (const r of results) {
    const q = r.quality || 'SD';
    const cap = caps[q] || 5;
    counts[q] = (counts[q] || 0) + 1;
    if (counts[q] <= cap) out.push(r);
  }
  return out;
}
```

Apply after ranking, before formatting. Target total: 25–35 streams for most queries.

## Exposing a Local Addon Externally

Some clients (Nuvio TV, web) need a publicly reachable URL, not a LAN IP.

### Tailscale Funnel

```bash
# Remove existing funnel first
sudo tailscale funnel reset

# Expose addon on port 3091
sudo tailscale funnel --bg 3091

# URL: https://{your-tailnet-hostname}.ts.net/
# Check status: tailscale funnel status
```

**Limitations**: One funnel per free plan. Use a reverse proxy (nginx/Caddy) if you need multiple services behind the same funnel.

### Cloudflare Tunnel (alternative, no Tailscale needed)

```bash
curl -L https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -o /tmp/cloudflared
chmod +x /tmp/cloudflared
/tmp/cloudflared tunnel --url http://localhost:3091
```

Gives `https://xxx.trycloudflare.com` — no account required.

### Nuvio TV Compatibility

Nuvio TV uses the same Stremio Addon Protocol as Stremio (manifest.json, /stream/:type/:id.json). Key differences:

- **Nuvio DOES NOT render iframe embeds**. Streams returning `url` that points to an embed page (VidSrc, SuperEmbed, 2Embed) MUST use `behaviorHints.notWebReady: true`. This tells Nuvio to attempt opening the URL in a WebView. However, many embed sites block iframe embedding, so server-side video URL extraction (via RC4 decrypt, seapi.link API, etc.) is strongly preferred.
- **Nuvio CANNOT play any embed stream that resolves to an iframe page only** — even with `notWebReady: true`, if the embed source returns an HTML page (not a direct video URL), Nuvio's ExoPlayer will fail with "None of the available extractors could read the stream (contentIsMalformed=false)". **Solution**: Only include embed streams whose extractors successfully resolve to direct video URLs (m3u8, mp4). If an embed extractor fails or returns iframe, exclude that stream entirely rather than serving a broken URL. **Remove sources that have NO extractor** (2Embed, EmbedSu, VidBinge) — they are always iframe and never work in Nuvio TV. If SuperEmbed's `seapi.link` API is unreachable (returns HTTP 000), also exclude it.
- **TorrServer proxy must wait for ready state before sending response** (NOT send empty headers immediately). The old "immediate headers" pattern caused Tailscale Funnel and upstream proxies to timeout (502/504) when no data arrived for 2-5s after the 200 response. **Correct pattern**: `waitForTorrent()` first → then get TorrServer stream → forward TorrServer's response headers (`content-type`, `content-length`, `accept-ranges`) + body to client. This way headers arrive simultaneously with video data. See `references/torrserver-proxy-nodejs.md` for the complete v3 pattern.\n- **TorrServer fileIdx=0 is NEVER valid** — TorrServer uses 1-based file indexing. The proxy must treat 0 as auto-detect and resolve to the correct video file ID from `file_stats`. See `references/torrserver-proxy-nodejs.md` for the auto-detect pattern.

- **Pre-warm TorrServer at scrape time**: When a search returns torrent results, proactively add the top 3 magnets to TorrServer via `prewarmTorrent(hash)`. This starts peer resolution before the user clicks play. With pre-warming, play requests hit TorrServer's cached metadata and stream in ~0.1s TTFB instead of 3-4s.

- **Auto-cleanup stale torrents**: Track access times per hash, remove from TorrServer after 120min idle (2 hours). Background interval every 5min checks access times. Keep torrents for longer than default 15min so users don't lose their stream mid-watch if they switch to another app briefly. Set `RemoveCacheOnDrop: true` in TS settings so disk cache is also deleted.
- **Torrent sources require debrid** in Nuvio as in Stremio. Without RD/TB/AD/PM API keys configured, infoHash streams will appear in the list but will not play.
- **Free P2P playback** (no debrid) requires a self-hosted TorrServer or Elementum. See `references/torrserver-proxy-nodejs.md` for the full v2 proxy with magnet polling, stat-code-based readiness detection, 12-tracker list, and streaming-optimized scoring.
- **Cinema HD / TeaTV / BeeTV use a fundamentally different architecture**: they run WebView-based JavaScript extraction on the Android device itself (not server-side HTTP scraping). They scrape streaming CDN sites (VidCloud, UpCloud, VidSrc) for direct .m3u8/.mp4 URLs, NOT torrents. This is why they have instant playback and no swarm downloads. See `streaming-source-aggregator` skill's `references/android-app-scraping-architecture.md` for the complete reference. For the full list of streaming sites they use, see `references/direct-http-streaming-sites.md`.

### Workflow: Inspect First, Then Code

When the user asks to add a new feature or change architecture, do NOT jump straight to implementation. First:

1. **Inspect** the repo — read the relevant source files, configs, and tests
2. **List** every file that will change and why
3. **Confirm** where config/API keys should be added (env vars, config JSON, etc.)
4. **Show** the user the plan before writing code

This prevents wasted work from incorrect assumptions about the codebase state.

## Optimization Techniques
Add `?preset=<name>` to stream URLs. Available presets in `default-config.json`:
- `torrentio-replacement` — Torrentio-only behavior
- `4k-enthusiast` — 4K priority, min 1080p
- `speed-demon` — 5s timeout, 20 max results
- `quality-over-quantity` — Best sources only
- `maximum-coverage` — All scrapers, full results
- `mobile` — No 4K, compact
- `anime` — Anime-optimized

## Speed Tiers
| `?speed=` | Scrapers | Timeout | Max | Use case |
|-----------|----------|---------|-----|----------|
| `fast` | Torrentio only | 5s | 30 | Quick preview |
| `balanced` | All enabled | 10s | 200 | Default |
| `comprehensive` | All + full | 15s | 500 | Power search |

## Metrics Endpoint

Two routes expose runtime metrics for monitoring and tuning:

**`GET /metrics`** — Prometheus text format (scrapeable by Grafana/Prometheus):
- `nexstream_uptime_seconds`
- `nexstream_requests_total`
- `nexstream_request_duration_ms{p50,p95,p99}`
- `nexstream_cache_l1_entries`, `nexstream_cache_l2_entries`, `nexstream_cache_l2_size_bytes`
- `nexstream_cache_hit_rate`
- Per-scraper: `nexstream_scraper_info{scraper="..."}` (status, requests, success rate, avg response, consecutive failures)

**`GET /metrics.json`** — JSON format with richer structure:
```json
{
  "server": { "version": "3.0.0", "totalRequests": 42 },
  "latency": { "p50": 0, "p95": 150, "p99": 400, "samples": 42 },
  "cache": { "l1": {...}, "l2": {...}, "hitRate": 62.5 },
  "scrapers": [{ "name": "torrentio", "status": "healthy", "avgResponseTime": 115 }]
}
```

Request times are tracked via `recordRequestTime(elapsed)` called from the stream route handler after each response (even cache hits). Rolling window of 1000 samples for percentile calculation.

## Docker Deployment
- Multi-stage build: `node:20-alpine` builder → production stage
- **Healthcheck:** Use `127.0.0.1` NOT `localhost` — inside Docker, `localhost` resolves to `::1` (IPv6) first, and apps binding to `0.0.0.0` don't listen on IPv6. Example: `wget --spider http://127.0.0.1:3091/health`
- Volume: `/app/src/data` for persistent SQLite cache
- Env vars for API keys at runtime (`docker run -e RD_API_KEY=...`)
- `--restart unless-stopped`

## Research Sources
When looking for features to adopt, check:
- `~/Unspooled-archive/` — saved competitor reference repos
- GitHub projects audited July 2026:
  - **moisa-addon** (peronecode) — Simple Stremio addon proxying Torrentio through TorrServer with **HTTP 302 redirect** instead of data proxy. Avoids bandwidth through the addon server entirely. Requires TorrServer directly reachable from client. 2 stars, Jan 2026.
  - **stremio-libtorrent-server** (andrewhack, 23 releases) — Full Python libtorrent engine + FFmpeg HW transcoding (NVENC/VAAPI). Adaptive piece picking plays head-first. Receives inbound peers for faster swarms. Runs as Docker, exposes full Stremio web player on port 8080/11470/12470. 3 stars, actively developed (July 2026). Replaces TorrServer entirely. NVENC requires `nvidia-container-toolkit` + Docker runtime config.
  - **NuvioStreamsAddon** (tapframe, 326 stars, archived) — Direct HTTP streaming for Nuvio TV, no torrents. Used **ScraperAPI** (paid) to bypass Cloudflare on streaming sites. Provider list: 4khdhub, hdhub4u, ShowBox, VidZee, uhdmovies, vidsrc, moviebox, hdrezkas, topmovies, dramadrip, anime sources. All sites now Cloudflare-blocked from server-side scraping. 343 commits, 22 releases. Archived Apr 2026.
  - **torrent-stream** (nyakaspeter) — Self-hosted Stremio addon, WebTorrent, in-memory storage, DuckDNS auto-TLS. TypeScript. 0 stars.
  - **stremio-torrent-stream** (nyakaspeter, 53 stars) — WebTorrent + Jackett, 2 years stale.
  - **Magnetio** (peterdsp, 19 stars, July 2026) — Self-hosted 22-provider Stremio addon with 8 debrid services, separate scraper+addon architecture, domain rotation with cooldown, bottleneck rate limiting, Redis caching, early return, content-based filtering. Most domains dead (403) as of July 2026. Infrastructure (httpClient, domainRotation, titleHelper, contentFilter) is MIT-licensed and worth borrowing. Full audit: `references/magnetio-audit-july-2026.md`.
- GitHub: YARR! Stremio, Comet, MediaFusion, AIOStreams, Torrentio
- Community: Tamtaro SEL templates, AIOStreams Discord, r/StremioAddons
- **Extractor catalog**: `references/scraper-engine-extractor-catalog.md` — 82 Kotlin video hoster extractors with porting recommendations
- **APK source**: `/home/rurouni/apk-analysis/scraper-engine/extractors/` — original Kotlin extractors (82 files)
- **TPB API is the only consistently working free source** (July 2026): Torrentio, Comet, and MediaFusion are all dead. Use `https://apibay.org/q.php?q={title}&cat=0` for free info hashes. **LimeTorrents** (`https://www.limetorrents.lol/search/all/{title}/seeds/`) is a reliable secondary — cheerio HTML parse with per-torrent magnet extraction. **BitSearch** (`https://bitsearch.to/search?q=...`) as tertiary. Most other public torrent APIs (YTS, 1337x, EZTV) are dead or aggressively blocked.
- **Unified Ultimate Scraper** (`~/ultimate-scraper/`, port 3091): production Node.js Express scraper that merges torrent, embed, and debrid sources into one service. As of July 2026: **12 torrent providers** (TPB, YTS, BitSearch, 1337x, EZTV, LimeTorrents, NyaaSi, TokyoTosho, AniDex, SubsPlease, AnimeTosho, Torznab), **6 embed extractors** (Filemoon ECDSA P-256, Rabbitstream AES-CBC, Vidsrc RC4/AJAX, Streamtape regex, VOE redirect, SuperEmbed multiembed.mov), 4 debrid providers, health monitor with circuit breaker (3 failures = degraded, 5 = down, exponential backoff), SQLite cache, and unified Stremio output (embed sources first, torrents second). RD cache check uses addMagnet→selectFiles→info poll (3×1.5s)→delete flow. The previous standalone free-scraper at `~/Scraper Suite/free-scraper/` has been merged into this service. For embed extractor techniques, see `~/ultimate-scraper/src/providers/embed/` or `streaming-source-aggregator` skill's `references/embed-extractor-techniques.md`. See `references/ultimate-scraper-reaudit-checklist.md` for the re-audit checklist and `references/multi-debrid-resolve-patterns.md` for provider API flows.
  - **APK target warning**: Ray clarified Ultimate Scraper is intended for use inside a streaming app APK, not only as a server-side service. Treat Express/Node routes as a test harness or optional LAN helper unless asked for backend deployment. For migration guidance, see `references/ultimate-scraper-apk-transition.md`.
  - **Simplified from legacy nexstream**: The old RTN engine, 11 formatter styles, progressive pipeline, external addon aggregation (Torrentio/Comet/MediaFusion), health monitor, speed tiers, and presets may be legacy depending on the current repo. Re-run a repo map and read current files before asserting which architecture is active.
- **Anime sources (5)**: Nyaa.si (cheerio parse — real titles/seeders/sizes), AniDex (JSON API), SubsPlease (JSON API), AnimeTosho (RSS HTML), TokyoTosho (cheerio HTML). All work server-side (no Cloudflare).
- **YTS API**: `https://yts.mx/api/v2/list_movies.json?query_term=...` — JSON API, frequently blocked server-side.
- **1337x**: Tries `torrent-api-py.onrender.com` proxy first, then falls back to direct HTML scrape of `1337x.to`. Multi-endpoint failover.
- **Free scraper (embed-based)**: `/home/rurouni/Scraper Suite/free-scraper/` — Node.js server with 6 ported extractors + Playwright headless + RC4 decryption, no debrid needed. Port 3092. Most extractors fail server-side against Cloudflare. Requires nodriver/FlareSolverr for reliable extraction.
- **Kodi addon architecture**: `references/kodi-addon-architecture-reference.md` — audited Fen Light+, Seren, Umbrella: debrid integration, scraper pipelines, caching, catalog backends (Pipelines section below has details)

## Go-Based Alternative (Non-Stremio)

For a full-stack Go scraper server (not Stremio-compatible), see `references/go-scraper-server.md`. This covers:
- Go Fiber server with TMDB catalog + 83 video hoster extractors
- Cloudflare bypass via nodriver (Python headless Chrome) — best 2026 option
- Porting Android/Kotlin scrapers to Go (TLS, crypto, regex differences)
- Streaming site frontend with iframe embed + hls.js playback
- Site-specific ID flow — the #1 mistake when porting Android scrapers. Native Android scrapers use site-internal slugs from their own search. Ported Go providers must also search the site first to get the correct slug — passing TMDB IDs directly never works.

## Web Movie Player (Python/Flask)

For a lightweight web movie streaming frontend with built-in TMDB search, The Pirate Bay API lookup, and TorrServer P2P playback, see `references/web-movie-player-flask.md`. This is useful as a Kodi-free testing interface or a simple mobile-friendly player.

### Architecture

```
Flask app (port 8800)
 ├─ TMDB API → search + movie details (metadata)
 ├─ TPB API (apibay.org) → info hashes (torrent sources)
 ├─ Jackett (opt) → Torznab indexers
 ├─ TorrServer (port 8090) → magnet → HTTP stream
 └─ Embed sources (VidSrc, 2Embed, EmbedSu, MoviesAPI, VidBinge) → instant iframe fallback
```

### Torrent Search Sources (no setup required)

**The Pirate Bay API** (free, no account, returns info hashes directly):
```
GET https://apibay.org/q.php?q={title}&cat=0
```
Returns JSON: `[{name, info_hash, seeders, size, ...}]`. The `info_hash` field is the 40-char hex BTIH. No registration needed. API is rate-limited but generous.

**Jackett** (requires self-hosted instance):
```
GET {host}:{port}/api/v2.0/indexers/all/results/torznab/api?apikey={key}&t=movie&q={title}
```
Returns Torznab XML with `<torznab:attr name="infohash">` elements. Requires installing Jackett binary and manually adding indexers via web UI at `/UI/Dashboard`.

### TorrServer Streaming Pipeline

### TPB API — No-Setup Torrent Source (Primary)

The Pirate Bay's JSON API (`apibay.org`) returns info hashes directly — no account, no configuration, free — and has replaced DMM/Zilean as the best zero-setup source:

## SanuFlix React Frontend (Alternative to Flask Templates)

For a production-quality web frontend that integrates with the Flask API backend, use the SanuFlix open-source React/Vite app (`github.com/Sanuu7/SanuFlix-opensource`):

1. **Clone** → `npm install` → configure `.env` with `VITE_TMDB_API_KEY`
2. **Configure** `src/config/servers.ts` with actual embed URLs (VidSrc, 2Embed, EmbedSu, etc.)
3. **Build** → `npm run build` → outputs to `dist/`
4. **Serve** via Flask: `send_from_directory('path/to/dist', 'index.html')`

The SanuFlix app already has: movie browsing, search, detail pages, genre filtering, server selector dropdown, Trakt-like UI, dark theme. Your Flask backend just needs to serve the static build and provide `/api/*` endpoints for TMDB search, torrent lookup, and TorrServer streaming proxy.

### Server Selector Config

The `STREAMING_SERVERS` array in `src/config/servers.ts` controls which embed sources appear in the UI dropdown. Each entry has `movieUrlTemplate` with `{tmdbId}` placeholder. The built-in server selector UI handles switching between sources seamlessly.

```
GET https://apibay.org/q.php?q={title}&cat=0
```

Response is a JSON array with fields: `name`, `info_hash`, `seeders`, `leechers`, `size`. Filter out entries where `seeders == 0` or `info_hash == '0000000000000000000000000000000000000000'`.

**Rate limiting**: Generous, no observed throttling at 1-2 req/s. TPB API has replaced the decommissioned DMM/Zilean public endpoint as the best zero-setup torrent hash source.

**Pitfall — default file_id is wrong**: Many torrents have non-video files first (subs, nfo, posters). The video is rarely file_id=0 or 1. **Always auto-detect** (see auto-detect pattern below).

**TorrServer binary** (MatriX fork, v141+):
- Download from GitHub releases, run standalone (no Docker needed if Docker has issues)
- Default port 8090, listens on 0.0.0.0
- **API endpoints** (MatriX fork specific):
  - `GET /echo` — health check, returns version string
  - `POST /torrents` — manage torrents: `{"action":"add","link":"magnet:...","save_to_db":true}`
  - `POST /torrents` — get/list: `{"action":"list"}` or `{"action":"get","hash":"..."}`
  - `GET /play/{hash}/{file_id}` — stream with auto-preloading (path format!)
  - `GET /stream/{hash}/{file_id}` — stream without preloading (path format!)
- **File stats**: response includes `file_stats` array with `{id, path, length}` for each file
- **Torrent states**: `Torrent added` → `Torrent getting info` (connecting to peers) → `Torrent working` (ready to stream)
- Peer discovery takes 10-30s for well-seeded torrents. Dead torrents never leave "getting info"

**⚠️ NOT `/api/torrents`**: The documented `/api/torrents` path returns 404 on the MatriX fork. Use bare `/torrents`.

**⚠️ NOT query params for streaming**: `/play?hash=X&file_id=Y` returns 404. Must use `/play/{hash}/{file_id}` path format.

**⚠️ Default file_id=0 or 1 is often wrong**: Many torrents have non-video files first (subs, nfo, posters). The video file is usually not file_id=0 or 1. Auto-detect by checking `file_stats` for the largest file with a video extension (`.mp4`, `.mkv`, `.avi`, `.mov`, `.webm`, `.m4v`).

- **TorrServer settings cannot be changed at runtime**: The POST `/settings` API on MatriX.142 is read-only (returns current values). Settings are stored in a BoltDB `config.db` created on first launch. To change settings: stop TS, delete `config.db`, write `settings.json` with desired values, then restart. TS creates a fresh `config.db` from `settings.json`. Optimized settings (4GB cache, 200 connections, upload cap 50MB/s, RemoveCacheOnDrop) are in `templates/deploy-libtorrent-server.sh`.

- **libtorrent-server alternative** (andrewhack): Full Python libtorrent engine with adaptive piece picking, NVENC hardware transcoding, and Stremio web player. Replaces TorrServer entirely. Deploy via `templates/deploy-libtorrent-server.sh`. API on port 11470 (HTTP), 12470 (HTTPS). Requires `nvidia-container-toolkit` for GPU passthrough.

### Auto-Detect Video File

When a torrent has multiple files (subs, nfo, poster, video), default file_id=0 or 1 may point to a non-video file. Auto-detect pattern:

```python
video_exts = ('.mp4', '.mkv', '.avi', '.mov', '.webm', '.m4v')
file_stats = torrent_data.get('file_stats', [])
video_files = [f for f in file_stats if any(f.get('path','').lower().endswith(e) for e in video_exts)]
if video_files:
    video_files.sort(key=lambda f: f.get('length', 0), reverse=True)
    file_id = str(video_files[0]['id'])  # largest video file
```

### Flask Proxy for CORS-Free Streaming

Browser `<video>` elements need `Accept-Ranges: bytes` and proper Content-Type. Direct TorrServer URLs may cause CORS issues in browsers. Flask proxy pattern:

```python
@app.route('/torrent/play')
def torrent_play():
    hash_val = request.args.get('hash', '')
    # auto-detect file_id from file_stats (see above)
    ts_url = f'http://{TS_HOST}:{TS_PORT}/play/{hash_val}/{file_id}'
    ts_resp = urllib.request.urlopen(ts_url, timeout=30)
    return Response(generate(), 200, headers={
        'Content-Type': 'video/mp4', 'Accept-Ranges': 'bytes',
        'Access-Control-Allow-Origin': '*',
    })
```

### JavaScript Player Polling

TorrServer needs time to connect to peers. The player should poll:

```javascript
// Poll torrent_status API for state changes
setInterval(function() {
    fetch('/api/torrent_status?hash=' + hash).then(r => r.json()).then(d => {
        if (d.stat_string === 'Working' && d.files > 0) {
            // Torrent ready, start playing
            video.innerHTML = '<source src="/torrent/play?hash=' + hash + '">';
            video.play();
            clearInterval(pollInterval);
        }
    });
}, 1000);

// Also try HEAD request as fallback
fetch('/torrent/play?hash=' + hash, {method: 'HEAD'}).then(r => {
    if (r.ok || r.status === 206) { /* stream ready */ }
});
```

### Embed Sources (always work, instant)

| Source | URL Pattern | Reliability |
|--------|-------------|-------------|
| VidSrc | `https://vidsrc.to/embed/movie/{imdb_id}` | ★★★★★ |
| 2Embed | `https://www.2embed.cc/embed/{imdb_id}` | ★★★★☆ |
| EmbedSu | `https://embed.su/embed/movie/{imdb_id}` | ★★★★☆ |
| MoviesAPI | `https://moviesapi.club/movie/{imdb_id}` | ★★★☆☆ |
| VidBinge | `https://vidbinge.dev/embed/movie/{imdb_id}` | ★★★☆☆ |

These are third-party streaming sites with ads. For mobile, embed works well in an iframe. For desktop, consider ad-blocking browser extensions.

### Key Differences from Go-Based Scraper Server

- **Python/Flask** — zero infrastructure, no compilation needed
- **TPB API** for info hashes (no Jackett/Prowlarr required)
- **TorrServer** for P2P streaming (not hoster scrapers)
- **Multiple embed fallbacks** — always work for popular titles
- **Mobile-optimized** Tailwind CSS, dark theme, responsive
- Blueprint architecture, deployable on any machine with Python 3

## Reverse-Engineering Existing Addons

When auditing Kodi addons for architecture patterns, API keys, and debrid dependencies:

### Finding Embedded API Keys

Grep patterns for any Kodi Python addon:
```
grep -rn "client_id\s*=\s*['\"][A-Za-z0-9]\{8,\}" --include='*.py'          # OAuth client IDs
grep -rn "api.key\|apikey\|api_key\s*=" --include='*.py' --include='*.xml'  # API keys
grep -rn "default.*key\|setting_default.*[A-Za-z0-9]\{20,\}" --include='*.py' # Defaults
grep -rn "X245A4XAIBGVM\|\d\{6,12\}['\"]" --include='*.py'                  # RD generic client
grep -rn "trakt.*client\|trakt.*secret\|tmdb.*apikey\|tvdb.*apikey" --include='*.py'
```

Key files to check (in order): `kodi_utils.py` → `globals.py` → `settings_cache.py` → `indexers/tmdb.py` → `settings.xml`

### Debrid Dependency Detection

Three commands tell you if an addon requires debrid:
1. `grep -rn "if not.*debrid\|prem_providers\|debrid_enabled\|No Debrid Services"` → hard gate
2. `grep -rn "resolvers\s*=\s*{"` → resolver list with only debrid = needs it
3. `grep -rn "torrentio\|comet\|knightcrawler"` → external scrapers return magnets → needs debrid

### Adding P2P Torrent Streaming (TorrServer)

To add P2P playback to a debrid-oriented addon without modifying scrapers:

**TorrServer API** (port 8090 default):
- `GET /echo` — health check (returns `uptime`)
- `POST /torrents` — `{"action":"add","link":"magnet:..."}` → returns `{"hash":"..."}`
- `GET /play?hash=...&file_id=N` → stream URL
- `GET /stream?hash=...&index=N&link=` → alternative stream URL

**Free torrent hash sources:**
- **DMM/Zilean**: `GET https://zilean.elfhosted.com/dmm/filtered?query=TITLE&Season=N&Episode=N` — **DECOMMISSIONED Jan 2026** (404/405). Was public, no account. Replace with self-hosted Zilean (Docker `ipromknight/zilean`) or a Jackett/Prowlarr instance.
- **Jackett**: `GET {host}:{port}/api/v2.0/indexers/all/results/torznab/api?apikey=...&t=movie&q=TITLE` — Torznab XML, parse for infohash. More reliable — 50+ indexers from a single API.
- **Prowlarr**: Same Torznab protocol as Jackett. Integrates with Arr stack for indexer sync.

**Integration pattern**: Scraper finds hashes → TorrServer `add_magnet()` → TorrServer returns stream URL → Kodi player.

### Scraper Interface Adapter

When combining scrapers from different addons, two interfaces exist:
- **Interface A** (Fen Light+): `class source → def results(self, info)` — uses window property for IPC
- **Interface B** (Umbrella): `class source` with class attrs `hasMovies`/`hasEpisodes`/`pack_capable` + `def sources(self, data, hostDict)` — returns list directly

Wrap Interface B in an adapter that calls `source.sources()` from within `def results()`. See `references/kodi-addon-architecture-reference.md` for full adapter code.

### P2P-Capable Addons (for non-debrid builds)

| Addon | Engine | Repo |
|-------|--------|------|
| **Elementum** | Built-in Go binary (libtorrent) | `github.com/elgatito/plugin.video.elementum` |
| **Jacktook** | TorrServer/Torrest/Elementum/Stremio addons | `github.com/Sam-Max/plugin.video.jacktook` |
| **TorrServer** | C++ standalone daemon | `github.com/yourok/torrserver` |

## Real-Debrid Integration

See `references/real-debrid-integration.md` for complete RD integration patterns.

## Cloudflare Bypass Options (2026)

Server-side Cloudflare bypass requires a real browser. uTLS/Go HTTP alone WILL be blocked.

### User preference: free-only solutions

This user's stack has **zero paid services**. No debrid subscriptions, no ScraperAPI, no premium proxies. Every solution must be entirely free, self-hosted, or use free-tier APIs (TMDB, TVDB). Paid options like ScraperAPI ($50/mo) must be marked clearly as last-resort alternatives.

### puppeteer-extra + stealth plugin (Node.js) — Free

```bash
npm install puppeteer-extra puppeteer-extra-plugin-stealth
```

Patches 18+ browser fingerprint vectors (navigator.webdriver, WebGL, plugins, languages). MIT licensed, free:

```js
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());

const browser = await puppeteer.launch({
  executablePath: '/usr/bin/chromium',
  args: [
    '--no-sandbox', '--disable-setuid-sandbox',
    '--disable-blink-features=AutomationControlled',
    '--window-size=1920,1080',
  ],
  headless: 'new',
});
```

Rotate viewport per page (`1920+rand(200) × 1080+rand(200)`) and block image/font/stylesheet loads to accelerate extraction. Stealth alone defeats basic Cloudflare but not JS challenge pages.

### Free alternative: nodriver (Python)

```bash
pip install nodriver
```

Successor to undetected-chromedriver. Direct Chrome DevTools Protocol, no Playwright middleware detectable. Best 2026 pass rate (28/31 sites tested).

### Summary (free options only)

| Tool | Cost | Pass Rate | Setup |
|------|------|-----------|-------|
| **puppeteer-extra + stealth** | Free | ~70% basic CF | `npm install`, 30s |
| **nodriver** (Python) | Free | ~90% | `pip install`, 30s |
| FlareSolverr | Free (Docker) | Good | Docker sidecar |

Start with `puppeteer-extra-plugin-stealth`. Add nodriver as Python subprocess fallback. ScraperAPI ($50/mo) exists but is NOT an option for this user.

**Implementation**: call as subprocess from Node/Go server when HTTP requests get blocked. Use session persistence for subsequent calls.

## Scraping Site-Specific IDs (Critical)

The #1 mistake when porting Android scrapers: use site-internal IDs, not TMDB IDs.

Native Android APKs (Streamflix, etc.) work because:
1. User searches WITHIN the site (e.g., Sflix search) → gets site-specific page URLs
2. Those URLs contain the site's own content IDs
3. IDs flow through: search → detail → servers → play

When porting to server code, replicate this flow:
```
TMDB title → search SITE for title → extract SITE's page URL/ID → get servers using site ID
```
Passing raw TMDB IDs directly to site providers returns 404s or homepage HTML.

## Common Pitfalls

- **`if (cached) return cached` — empty arrays are truthy**: `[]` is truthy in JavaScript. If a previous scrape cached `[]` (e.g., all provider domains were dead and returned nothing), every subsequent request returns `[]` immediately without running any providers. The pipeline appears broken until the cache is cleared. **Fix**: Check `cached !== null && cached !== undefined && cached.length > 0` before returning from cache, or use `if (cached && cached.length)`. Also clear the persistent SQLite cache on deployment changes: `rm -f data/cache.db`.

- **Content filter `isTorrentMetadata` excludes year numbers**: The pattern `/^\d/` in `isTorrentMetadata()` catches ANY word starting with a digit — including year numbers like "2010". This means single-word movie titles like "Inception" never meet the minimum word-count requirement because "2010" is excluded from the match count. **Fix**: Remove `/^\d/` from `isTorrentMetadata`. Years are search terms, not torrent metadata.

- **Content filter minimum word count too aggressive for single-word titles**: `Math.max(2, Math.ceil(searchWords.length * 0.7))` requires at least 2 matching words, but single-word titles like "Inception" or "Gladiator" can only match 1. **Fix**: Use `Math.max(1, ...)` so single-word searches accept 1 match. Multi-word titles still need ≥70%.

- **Content filter `required` count includes stop words (the, a, an)**: The `required` count is calculated from ALL `searchWords` including stop words like "the", "a", "an", but `wordMatchCount` excludes those via `isTorrentMetadata()`. For "The Lion King": `searchWords = ["the", "lion", "king"]` (3 words), `wordMatchCount` can reach at most 2 (lion+king) since "the" is metadata. If `required = ceil(3 * 0.7) = 3`, the filter ALWAYS fails — zero streams pass. **Fix**: compute `required` from content-bearing words only, excluding stop words: `const contentWords = searchWords.filter(w => !isTorrentMetadata(w)); const required = Math.max(1, Math.ceil(contentWords.length * 0.7));`. This way "The Lion King" requires 2 of 2 content words, and "Inception" requires 1 of 1.

- **Magnetio provider domains are mostly dead/403**: The 22 Magnetio providers' domain lists were captured in March 2026. By July 2026, most return 403 (TorrentGalaxy, 1337x, EZTV, YTS, KickassTorrents, etc.). Only TPB via `apibay.org` consistently works. Domain rotation with fallback domains helps but many fallback domains are also dead. **Strategy**: Add new providers with fresh domain testing. Maintain a working-provider whitelist alongside the full provider registry.

- **Magnetio provider registration uses named exports, not default**: Magnetio providers use `export const id`, `export const name`, `export async function scrape`. When importing, use `import * as prov from './prov.js'` (namespace import), not `import prov from './prov.js'` (default import). Using the wrong import type causes `SyntaxError: does not provide an export named 'default'`.

- **Magnetio `buildSearchQuery` may return unrelated results**: TorrentGalaxy's search ignores the `search` parameter in some cases, returning 190 latest-uploaded results (e.g., "The Death Of Robin Hood 2026" for "Inception 2010"). The content filter then drops everything as unrelated. **Fix**: Verify search results contain the expected title before trusting a provider. Some sites require different parameter formats (cookie, session-based searches).

- **MKV → browser playback fails**: Real-Debrid unrestricted links point to MKV files (especially REMUX/BLUriP sources). The browser's `<video>` element only supports MP4/WebM/OGG. MKV container format is not supported in any browser.
  - **Fix**: Run the stream through ffmpeg with `-c copy -movflags frag_keyframe+empty_moov -f mp4 pipe:1` to transmux MKV → fragmented MP4 in real-time. This copies video/audio bitstreams (no re-encode, near-zero CPU). Add ffmpeg to Docker: `apk add --no-cache ffmpeg`.
  - If codec is still unsupported (AV1, HEVC on Chrome), fall back to direct RD link with "Open in VLC" instructions.
  - See `references/ffmpeg-stream-proxy.md` for a production-ready Node.js stream proxy route.

- **Title matching too loose — wrong movies leak through**: A search for "The Matrix" returns "The Matrix Revolutions" because title substring similarity is too permissive.
  - **Fix**: Implement `isWrongMovie()` check — if ALL words from the context title match but the result has extra words that are sequel indicators (ii, iii, reloaded, revolutions, part 2, etc.), reject the match.
  - **Strict thresholds**: Require 0.85 similarity for movies (not 0.8), 0.6 for series (not 0.5). Match year ±1. If year differs by >2, reject entirely.

- **Sequel/prequel detection**: Build a list of sequel indicator tokens (`/\bii\b/, /\biii\b/, /\breloaded\b/, /\brevolutions\b/, /\breturns\b/, /\brevenge\b/, /\bpart\b/, /\bchapter\b/`). If result title has all context title words plus sequel words, it's the wrong movie. Exception: season packs should pass (they have S01-S05, not sequel words).

- **DSU dedup too aggressive**: Don't union streams sharing common words like title names. Use resolution-aware DSU (only dedup same resolution tier).
- **Debrid keys on server**: User's debrid keys should stay in the app. Server-side checking is optional and uses separate keys.
- **Test credentials: never persist to .env or config files**: When the user gives you an API key for testing, pass it at container runtime via `-e KEY=value` (docker run) or inject via terminal for a single curl test. Do NOT write test keys to `.env`, `default-config.json`, or any file that ends up committed. The user will push back: "I didn't give you permission to hardcode my key." Test keys are test-only.
- **RD API keys with `!` chars**: Real-Debrid API keys contain `!` which bash interprets as history expansion. Always wrap in single quotes: `-e 'RD_API_KEY=...'` or use `set +H` before the command.
- **Re-auditing recommended fixes**: When the user asks to re-audit prior scraper fixes, verify each recommendation against current code and live endpoints. Use `references/ultimate-scraper-reaudit-checklist.md` for the exact Ultimate Scraper checks: `.env`/TMDB state, RD cache strategy, `/api/resolve`, 1337x fallback, BitSearch parser, version consistency, tests, and health endpoints.
- **Cache key must include format**: When implementing formatter styles, append `:fmt=<style>` to the cache key. Otherwise all format requests return the same cached response.
- **Progressive cache key collision**: Progressive mode uses TWO cache keys: `cacheKey` (full results, long TTL) and `cacheKey + ':bg'` (progressive partial results, 30s TTL). Background fill clears the `:bg` key when the full cache is set.
- **Docker volume persistence**: SQLite cache survives container restarts in Docker volumes. Clear with `docker volume rm scraper-cache` for fresh tests.
- **Cross-project research**: Analyze ALL projects in the archive before implementing. Don't stop at the first one.
- **JS `name` variable shadowing**: When destructuring `const { name } = stream` and then declaring `let name = '';` in the same function scope, you get a redeclaration error. Rename the destructured property: `const { name: streamName } = stream` and use `displayName` for new variables.
- **Destructured variable name mismatch (`data` vs `html`)**: When using `const { data: html } = await http.get(...)`, the response is renamed to `html`. A subsequent reference to `data` — e.g. `JSON.parse(data)` or `parsed = data` — throws `ReferenceError: data is not defined`. Always verify every reference in the function body matches the destructured name, not the original key. This bug affected the SubsPlease provider for months.
- **Shell pipes and `set +H`**: When passing credentials through shell pipes, bash history expansion (`!`) can corrupt values. Use `set +H` to disable, or use single-quoted heredocs.
- **`runSingleScraper().catch()` crash**: Calling `.catch()` directly on an async function's return can throw `Cannot read properties of undefined (reading 'catch')` if the function errors synchronously before returning a Promise. Fix: wrap in `(async () => { try { return await fn(); } catch { return []; } })()` instead — this also properly catches synchronous exceptions that `.catch()` can't handle.
- **Headless browser first-call latency**: First nodriver/chromedp subprocess call is 3-5s (Chrome startup). Use session persistence or warm-up mechanisms for production.
- **RD `checkCache` / `instantAvailability` is brittle — verify current docs first**: Do NOT state that `/torrents/instantAvailability` is absent from official docs unless you have checked `https://api.real-debrid.com/` in the current session. In July 2026 the official docs still listed `GET /torrents/instantAvailability/{hash}`. The durable failure mode is silent false negatives: code that catches all RD errors and returns `false` makes cache badges lie. Safer options: log RD API error code/message and mark status unknown, or use addMagnet → selectFiles → poll info → if status==="downloaded" it was cached → unrestrict. For provider-specific flows, see `references/multi-debrid-resolve-patterns.md`.
- **Test API keys: never persist**: When testing with a user's debrid API key, pass at container runtime via `docker compose build` + `up -d` after writing the `.env`. Never commit to the repo. User will push back if key ends up in version-controlled files.
- **Embed/iframe streams MUST use `behaviorHints.notWebReady: true`**: Setting `notWebReady: false` (or omitting it) tells the client "this URL is a direct video stream." The client will try to open embed URLs in a native video player and fail. Always set `notWebReady: true` for embed/iframe URLs. Do NOT use a custom `embedType` field (e.g. `embedType: 'iframe'`) — no Stremio-compatible client recognizes custom fields.
- **Missing `fileIdx` causes playback failures on some clients**: Always seed `fileIdx: 0` as default in the provider-level `buildResult()` so every stream has it. Don't rely on the route handler to add it — if a stream object comes from an external source or custom provider without `fileIdx`, the client may refuse to play it.
- **Seeders MUST be factored into scoring for non-debrid streaming**: The default `scoreStream()` function weights resolution (4K=100), quality (REMUX=100), encode (HEVC=90), HDR/audio tags, and match score — but entirely ignores seeders. For debrid users, seeders don't matter (debris serves cached data). For TorrServer/P2P streaming, seeders are THE critical metric — a 1080p YIFY with 642 seeders will play instantly, while a 4K REMUX with 5 seeders may never start. **Fix**: Add a seeder bonus to `scoreStream()`:

```js
const seeders = item.seeders || 0;
if (seeders >= 500) score += 80;
else if (seeders >= 100) score += 60;
else if (seeders >= 50) score += 40;
else if (seeders >= 20) score += 20;
else if (seeders >= 10) score += 10;
else if (seeders >= 5) score += 5;
```

This pushes well-seeded 1080p/720p releases above high-quality but dead 4K REMUX streams.

- **Audio-only releases (music albums, OST) bypass content filter**: Torrents like "[Mashin] INCEPTION 1stアルバム[320K]" or "OST Inception FLAC" pass the content filter because "INCEPTION"/"Inception" appears in the title. The metadata matcher then accepts them as title matches. **Fix**: Add audio/video indicator detection before movie matching:

```js
const audioIndicators = /(320[kK]|flac|mp3|aac|album|ost|soundtrack|discography|320kbps|v0\b|v2\b)/i;
const videoIndicators = /(x264|h264|h265|hevc|x265|bluray|1080p|2160p|720p|web-dl|webrip|bdrip|dvdrip|remux|hdtv)/i;
if (audioIndicators.test(title) && !videoIndicators.test(title)) {
  return { match: false, reason: 'Audio-only release', score: 0 };
}
```

Also clear the SQLite cache (`rm -f data/cache.db`) when deploying scoring changes — stale caches serve old-scored results.

- **TorrServer PreloadCache default causes 30-60s buffering before playback**: TorrServer's default `PreloadCache: 90` setting (in `settings.json`) means it tries to preload **90% of CacheSize** before serving any video data. With the default 64MB cache, that's ~57MB preload (fast). But if CacheSize was bumped to 4GB, PreloadCache=90 means **3.6GB preload** before the first byte is served — the entire video plays "please wait" until 3.6GB downloads. **Fix**: Set `PreloadCache` to **5** (5% of cache, ~50MB with 1GB cache). Playback starts in 1-3s instead of 30-60s. Settings file location and structure:

```json
{
  "BitTorr": {
    "CacheSize": 1073741824,    // 1GB — enough for smooth streaming
    "PreloadCache": 5,          // 5% — download 50MB then start playing
    "ReaderReadAHead": 95,      // keep default — forward-read focused
    "ResponsiveMode": true,     // keep default
    "UseDisk": false,           // RAM cache (faster than disk)
    "ConnectionsLimit": 500,    // more peer connections
    "RemoveCacheOnDrop": true,  // cleanup on torrent removal
    "StoreSettingsInJson": true // use JSON file for settings
  }
}
```

The settings file is at the `--path` directory: `{path}/settings.json`. TorrServer MatriX.142 reads this JSON on startup when `StoreSettingsInJson: true`. To apply changes: edit the file, kill TorrServer (`kill $(pgrep torrserver)`), restart. No need to delete `config.db` — TS reads JSON at startup.

**Key insight**: The `Preload` function in TorrServer's `apihelper.go` computes:
```go
size = int64((cacheSize / 100.0) * preloadCache)
```
With CacheSize=4GB (4,294,967,296) and PreloadCache=90: `size = 42,949,672.96 * 90 = 3,865,470,566` (3.6GB). That's the entire download. 1080p YIFY files (2-3GB) would be **fully downloaded** before any video played.

**Optimal values for streaming** (tested on RTX 2080 Ti server with 11GB VRAM + 32GB RAM):
- CacheSize: 1GB (1073741824) — more than enough for smooth streaming
- PreloadCache: 5 (only 50MB pre-buffer, starts playing immediately)
- UseDisk: false (RAM is faster; server has abundant memory)
- ConnectionsLimit: 500 (more peers = faster swarm join)
- ReaderReadAhead: 95 (aggressive forward-read for sequential play)
- ResponsiveMode: true (prioritize playback responsiveness)

For a complete reference on TorrServer cache/storage internals, see `references/torrserver-settings-cache-tuning.md`.

- **`infoHash` + `url` conflict causes "red addon" in Nuvio**: When a stream object has BOTH `infoHash` AND `url` fields, some clients (Nuvio, some Stremio forks) try to handle the torrent internally instead of using the provided direct URL. If the client can't torrent (no libtorrent, no debrid), the stream fails and the addon gets marked broken/red. **Fix**: When providing a TorrServer (or any direct play) URL, delete `infoHash` and `fileIdx` from the stream object so the client has no choice but to use the URL:
  ```js
  stream.url = `https://${req.get('host')}/api/torr/play/${hash}/${fileIdx}`;
  stream.isFree = true;
  delete stream.infoHash;
  delete stream.fileIdx;
  ```
  Only keep `infoHash` on streams WITHOUT a direct URL (fallback torrents).

- **Early return still blocks on slow providers via `Promise.allSettled`**: The common pattern `if (resultsArrays === 'EARLY') { const partial = await Promise.allSettled(promises); ... }` looks like it returns after the early-return timer fires, but `Promise.allSettled` still waits for ALL pending provider promises to settle (up to their individual 12s timeouts). The timer fires at 500ms/1s, then the next line blocks again for 11s more. **Fix**: Use a shared array that each provider pushes results into as they complete. On early return, read whatever is in the array — no waiting:
  ```js
  const rawAllResults = [];
  const promises = providers.map(async (prov) => {
    // ... scrape ...
    rawAllResults.push(...tagged); // push as they complete
  });
  await Promise.race([
    Promise.allSettled(promises),
    new Promise(r => setTimeout(r, EARLY_RETURN_MS)),
  ]);
  // rawAllResults already has everything — continue immediately
  ```
  This cuts first-request response time from 16s to ~1.5s (500ms early return + 1s processing). Cached requests still hit ~0.75s.

- **Slow embed fetch and Chromium scraping block the response**: After the scrape pipeline returns results, `buildEmbedStreams()` (HTTP requests to multiembed.mov + vidsrc.to, ~2-3s each) and `scrapeMovieDirect()` (Chromium launch, ~3-10s) are awaited sequentially. This adds 5-15s to the response after torrents are already ready. **Fix**: Run embed and Chromium scrapers in parallel with 500ms timeouts, so torrent streams return immediately:
  ```js
  const embedPromise = buildEmbedStreams(context).catch(() => []);
  const directPromise = scrapeMovieDirect(...).catch(() => []);
  const [embedStreams, directResults] = await Promise.all([
    Promise.race([embedPromise, new Promise(r => setTimeout(() => r([]), 500))]),
    Promise.race([directPromise, new Promise(r => setTimeout(() => r([]), 500))]),
  ]);
  ```
- **"Addon appears then goes red" diagnostic**: This exact symptom has four likely causes, in order of frequency:
  1. **`infoHash` + `url` conflict** (see above) — client tries internal torrent, fails
  2. **Manifest field errors** — three specific fields that MUST be in the manifest for Nuvio compatibility:
     - `catalogs: []` — MUST be an empty array, not missing. Some clients crash reading `.length` on `undefined` when they enumerate catalogs.
     - `behaviorHints.p2pNotSupported: true` — tells the client "I handle all streaming URLs directly, don't initialize P2P." Without this, the client may spawn a torrent subsystem, fail, and mark the addon broken.
     - `behaviorHints.adult: false` — required protocol field.
  3. **Catalog endpoint fails** — manifest declares `catalog` resources but the endpoint times out, returns empty, or the user doesn't want catalogs. Remove `catalog` from `resources` array entirely and set `catalogs: []` if the addon doesn't need to provide Discover catalog data. The TMDB key is for metadata matching during scraping, not for catalog serving.
  4. **Client-specific timeout** — some clients (Nuvio on weak hardware, mobile) have aggressive per-call timeouts (5-8s). Stream endpoint returning >15s on first hit triggers addon failure. **Fix**: add per-response timeouts for slow embed fetch and Chromium scraping (500ms each), use shared-array early return (500ms first batch, fallback to 6s max), and process in parallel not series.
  
  **Workflow**: (a) test manifest externally `curl -s -o /dev/null -w "%{http_code}" <funnel-url>/manifest.json`, (b) check manifest has `catalogs: []` and `p2pNotSupported: true`, (c) test stream externally and measure response time `curl -s -o /dev/null -w "%{http_code} %{time_total}s" <funnel-url>/stream/movie/tt1375666.json`, (d) inspect first stream object `curl -s <funnel-url>/stream/movie/tt1375666.json | python3 -c "import json,sys;d=json.load(sys.stdin);s=d['streams'][0];print('url' in s,'infoHash' in s)"` — if both are `True`, that's cause #1.\n\n- **`MediaCodecVideoRenderer` error with `NO_EXCEEDS_CAPABILITIES` — device can't decode HEVC**: Many Android TV devices (especially budget boxes, older SoCs, or Fire TV sticks) lack hardware HEVC (h265) decoders. When the first stream offered is HEVC, ExoPlayer fails with `format supported=NO_EXCEEDS_CAPABILITIES` and the user sees \"This stream uses a format your device may not support.\" **Root cause**: The default `ENCODE_SCORE` ranks HEVC=90, H264=70 — opposite of what these devices need. **Fix chain**:\n\n  1. **Reverse `ENCODE_SCORE`** in `metadataMatcher.js` to prefer AVC:\n     ```js\n     const ENCODE_SCORE = {\n       'HEVC': 50,       // Poor HW decode on many Android TV devices\n       'AV1': 30,        // Virtually no HW decode\n       'H264': 100,      // Widest HW decode support\n       'AVC': 100,\n       'VP9': 40,\n       'XviD': 20,\n       'DivX': 15,\n       'Unknown': 30,\n     };\n     ```\n\n  2. **Add codec penalty** in `scoring.js`'s `scoreResult()` — a direct negative modifier for HEVC/AV1 that doesn't depend on the encode score weight:\n     ```js\n     function getCodecPenalty(codec) {\n       if (!codec) return 0;\n       const name = codec.toUpperCase();\n       if (name === 'HEVC' || name === 'H.265' || name === 'X265') return -60;\n       if (name === 'AV1') return -80;\n       if (name === 'H264' || name === 'AVC' || name === 'X264') return 0;\n       return -10;\n     }\n     // In scoreResult():\n     if (c) {\n       const codecScore = CODEC_STREAMING_RANK[c] || 50;\n       score += codecScore * 0.15;\n       score += getCodecPenalty(c);  // Add heavy penalty for HEVC/AV1\n     }\n     ```\n\n  3. **Add hard filter at TorrServer URL selection** — when AVC alternatives exist, skip HEVC/AV1 from the top TorrServer playable slots. This ensures the first clickable stream always works:\n     ```js\n     // Check if any AVC stream exists in results\n     const hasAvcAlternative = capped.some(r => {\n       const parsed = r.parsed || parseRelease(r.title || '');\n       const codec = parsed?.codec || r.codec || '';\n       return ['H264', 'AVC', 'X264'].includes(codec.toUpperCase());\n     });\n\n     // In the stream building loop:\n     const isBadCodec = codec && ['HEVC', 'X265', 'H.265', 'AV1'].includes(codec.toUpperCase());\n     const canUseTorrServer = idx < 10 && r.infoHash && hasEnoughSeeders && (!hasAvcAlternative || !isBadCodec);\n     ```\n\n  4. **Include `codec` in stream objects** so the user can see before clicking:\n     ```js\n     stream.codec = parsed?.codec || r.codec;\n     ```\n\n  **Verification**: After the fix, the first 3-5 playable streams should all be AVC/H264 with TorrServer URLs. Test with:\n  ```bash\n  curl -s \"https://addon-url/stream/movie/tt0111161.json\" | python3 -c \"\n  import json,sys; d=json.load(sys.stdin)\n  for s in d['streams'][:5]:\n      print(f\"{s.get('codec','?'):5} {s.get('seeders',0):3}👤 {s.get('quality')}\")\n  \"\n  ```\n  Expected output — all first 5 entries show `H264`, not `HEVC`.\n\n  **Caveat**: When NO AVC streams exist (rare anime or obscure content), HEVC/AV1 should still be offered as a last resort. The `hasAvcAlternative` guard handles this: if no AVC exists, HEVC gets TorrServer URLs anyway. Also clear the SQLite cache (`rm -f data/cache.db`) when deploying scoring changes — stale caches serve old-scored results.`

## Key Files

- `src/pipeline/orchestrator.js` — Pipeline controller; `execute()` (sync), `executeProgressive()` (progressive), `processStreamBatch()` (reusable batch processor)
- `src/pipeline/rtn.js` — RTN engine: release group extraction (150+ groups), tier detection, proper/repack/internal, year, language
- `src/pipeline/formatter.js` — 11 display styles (torrentio, minimal, standard, detailed, compact, netflix, debrid, info, groups, simple, noEmoji)
- `src/pipeline/ranker.js` — Scoring engine with RTN group bonuses
- `src/routes/stream.js` — Route handler: speed tiers, presets, format param, progressive toggle
- `src/scrapers/external-addon.js` — Generic Stremio addon fetcher for aggregation
- `src/pipeline/health-monitor.js` — Circuit breaker + EMA adaptive timeout
- `src/http-client.js` — Keep-alive connection pooling
- `src/cache.js` — L1 in-memory (NodeCache) + L2 SQLite (better-sqlite3)
- `src/config/default-config.json` — Full config: scrapers, debrid, filtering, RTN ranking, limits, presets, externalAddons, format styles, performance
- `references/kodi-addon-architecture-reference.md` — Kodi addon arch patterns: debrid integration, scraper pipelines, interface adapters, API keys, TorrServer P2P integration, combined addon builds, **web search UI for testing without Kodi**, **Kodi mock test harness patterns**
- `references/ffmpeg-stream-proxy.md` — Production-ready Node.js stream proxy for MKV->fragmented MP4 transmuxing
- `references/free-embed-scraper.md` — Free scraper server (Node.js, 6 embed extractors, no debrid)
- `references/stremio-stream-formatter-patterns.md` — Complete stream `name` + `description` formatter with resolution badges, cache indicators, HDR type detection, quality tags, and full integration pattern
- `references/aiostreams-config-to-code-mapping.md`
- `references/torrserver-settings-cache-tuning.md` — TorrServer settings.json structure, PreloadCache tuning, CacheSize, optimal streaming values — Field-by-field mapping of AIOStreams Tamtaro SEL v2.6.1 config to equivalent server-side implementations in a self-hosted scraper
- `references/nvenc-gpu-passthrough.md` — NVENC passthrough for libtorrent-server Docker (nvidia-container-toolkit install, Docker runtime config, deploy commands, TRANSCODING_MODE env vars)
- `references/metadata-matching-reference.md` — Full reference for TMDB+TVDB matching, strict title/year/SE matching, reboot/sequel detection, scoring weights table, conditional filtering, TVDB v4 client, stream formatter example output

### Ultimate Scraper Reference Implementation

The working reference implementation lives at `~/ultimate-scraper/` (port 3091):

| File | Purpose |
|------|---------|
| `src/core/streamFormatter.js` | `formatStream()`, `buildStreamName()`, `buildStreamDescription()` |
| `src/core/smartDedup.js` | `deduplicate()` with 3-key strategy (infoHash, filename, fingerprint) |
| `src/core/metadataMatcher.js` | `matchResult()` with wrong-movie detection and strict similarity |
| `src/core/scoring.js` | `rankResults()` with cached/uncached split sort + HDR/audio boosts |
| `src/core/releaseParser.js` | `parseRelease()` with HDR type, IMAX, audio channels extraction |
| `src/routes/scrape.routes.js` | Stremio `/stream/:type/:id.json` route with full pipeline |
| `src/services/externalAddonFetcher.js` | Fetch streams from external Stremio addons (Torrentio, Peerflix) |