# Scraper Pipeline Research — Competitive Architecture Analysis

## AIOStreams (SEL-based)

- **SEL (Stream Expression Language):** Custom DSL with lexer/parser/AST/evaluator — boolean logic on stream attributes (`quality=4k AND cached=True`)
- **Config Template System:** JSON template with 89+ config keys — services, presets, formatters, filters, ranking, synced blocklists
- **Debrid:** Adapter/Strategy pattern — `BaseDebrid` abstract class, concrete adapters per provider, factory/registry for selection
- **Scraper Pipeline:** ETL — Extract (fan-out to 5+ services) → Transform (normalizer per source) → Dedup (merged by infoHash) → Filter (SEL evaluator) → Rank (user priority)
- **Cache:** Hierarchical multi-tier (L1 in-memory, L2 Redis, L3 SQLite) with `@cached(ttl=...)` decorator pattern
- **Parallel Fetching:** `asyncio.gather()` with per-service `asyncio.Semaphore` for rate limiting
- **Key takeaway:** The SEL DSL is the most powerful user-facing filtering system in the Stremio ecosystem

## YARR! Stremio (typescript, 48+ scrapers)

- **Architecture:** Express + Stremio Addon SDK, TypeScript ESM
- **Scrapers:** 48+ site scrapers (Cheerio-based for HTML, RSS/API for structured sources, two-pass for magnet extraction)
- **Smart Dedup:** 4 strategies (infohash, tracker+name, smart-hash via size+resolution+codec, filename)
- **RTN Ranking:** Sophisticated torrent name scoring — trusted release groups (SPARKS, RARBG, YTS), log-scale seeders, codec/audio/HDR bonuses, size penalties
- **Health Monitor:** Tracks success/failure per scraper, auto-disable after 3 consecutive failures, periodic retry after 30 min, EMA response times
- **Request Queue:** In-flight dedup — concurrent same-ID requests merge into one promise
- **Speed Tiers:** Fast (8s/5 results), Balanced (20s/10), Comprehensive (45s/20) — configurable scrapers and timeouts per tier
- **Multi-Debrid:** 6 services (RD/AD/PM/TB/DebridLink/PikPak) with fallback chain (direct API → StremThru proxy → individual)
- **ML Engine:** Learns user preferences from selections — qualities, trackers, codecs, sizes
- **Provider Presets:** content-type presets (movies/tv/anime/international) auto-enable relevant scrapers
- **Formatters:** 11 display styles (yarr, torrentio, minimal, detailed, compact, pro, etc.)
- **Key takeaway:** Best-in-class dedup, ranking, and health monitoring; massive scraper ecosystem

## Comet (Python/Starlette)

- **Architecture:** FastAPI/Starlette, async Python, Stremio addon protocol
- **Pipeline parallelism:** Checks debrid cache status CONCURRENTLY with scraping — eliminates a full network round-trip
- **Caching:** Native V8 `Map` objects for L1 hash lookups
- **Debrid:** Batch instantAvailability checks, optimized streaming URL generation
- **Key takeaway:** Pipeline parallelism (debrid check concurrent with scrape) is the single biggest latency optimization

## MediaFusion (Python)

- **Architecture:** FastAPI, async Python, extensive caching
- **Caching:** Heavy Redis usage for shared cache; proactive cache daemon watches user's RD cloud library and pre-populates
- **Streaming responses:** Pushes results as they arrive via streaming JSON — addon feels instant
- **Debrid:** Batches infoHash lookups to minimize API calls; native cloud sync
- **Key takeaway:** Streaming JSON responses and proactive cache warmup

## DebridStream (watchlist-driven)

- **Architecture:** Async task queue (APScheduler/Redis) processing watchlist items
- **Plugin system:** Scrapers loaded dynamically from `config.yaml` — users add/remove sources without touching core
- **Quality Scoring:** Heuristic metadata parsing (guessit/trakit) + user-defined quality profiles with scoring thresholds
- **Dedup:** Multi-layer stateful dedup (content-level by IMDB/TMDB, file-level by infoHash, debrid-level by instantAvailability)
- **Symlink farming:** Zero-local-storage — files symlinked to rclone/Zurg mount of debrid cloud
- **Key takeaway:** Plugin-based scraper architecture + zero-local-storage streaming

## nexstream-scraper v2 Architecture (our implementation)

```
Request → PARALLEL SCRAPE (3 scrapers: Torrentio/Comet/MediaFusion)
         → NORMALIZE (standard Stream schema)
         → DSU DEDUP (resolution-aware Disjoint Set Union)
         → DEBRID ENRICH (optional, server-side keys)
         → ADAPTIVE FILTER (CAM/TS/HC exclusion, quality-tier adaptive)
         → RANK (resolution weight + quality bonuses + cached bonus)
         → RETURN (Stremio-formatted, cached in L1 memory + L2 SQLite)
```

Features implemented:
- **Config-driven:** `default-config.json` with scrapers, debrid, filtering, ranking, limits, performance sections
- **Quality detection:** Regex-based (4K, HDR, DV, Atmos, TrueHD, REMUX, CAM, TS, etc.)
- **DSU Dedup:** resolution-aware — never merges 4K with 1080p
- **Health monitor:** Per-scraper success/failure tracking, auto-disable after 3 failures, 30-min retry
- **Request queue:** In-flight dedup for concurrent duplicate requests
- **Speed tiers:** fast (Torrentio only, 5s) / balanced (all, 10s) / comprehensive (all + full, 15s)
- **Keep-alive pooling:** Shared `http.Agent` with `keepAlive: true` across all scrapers
- **Debrid availability cache:** 5-min TTL per hash per provider
- **Connection pre-warming:** HEAD requests to scraper endpoints at startup