# APK Scraper Migration Audit

Bottom line: keep the scraper orchestration inside the APK, but do **not** embed the current Node/Express service directly. Port the small torrent-source subset to Kotlin first, keep cloud/browser-protected extractors behind an optional helper, and feed results into the existing `ScraperLink` → `DebridRouter` path.

## Evidence

| Area | Evidence |
|---|---|
| Existing Android entry point | `app/src/main/java/com/unspooled/app/data/scraper/StreamScraperEngine.kt` already scrapes config-driven endpoints and routes through `DebridRouter`. |
| Existing Android scraper config | `DefaultAddonScrapers.ALL` is currently empty, so the embedded scraper provider is disabled. |
| Existing Android data model | `ScraperLink` supports `infoHash` and direct `httpUrl`; this matches the useful output shape from Ultimate Scraper. |
| Existing debrid flow | `DebridRouter` already checks cache and resolves info hashes through `DebridService`. |
| Existing RD behavior | Android `RealDebridClient.getDownloadUrl()` already uses `addMagnet` + poll + `selectFiles`, not the brittle `instantAvailability` server path. |
| Current Node output | Ultimate Scraper `/api/scrape` returns `{ infoHash, title, source, quality, size, seeders }`, which maps cleanly into `ScraperLink`. |
| Android dependencies | App already has OkHttp, Retrofit, kotlinx-serialization, Gson, Room, Media3. It does **not** have a DOM parser equivalent to Cheerio yet. |

## What should be embedded directly in the APK

| Component | Embed? | Reason | Kotlin target |
|---|---:|---|---|
| Result schema | Yes | Small plain model; already mostly exists as `ScraperLink`/`Stream`. | Add `TorrentSearchResult` internal model only if needed. |
| Metadata context building | Mostly yes | App already has TMDB/Trakt infrastructure; avoid calling Node just to resolve title/year. | Reuse existing app metadata layer, not `services/tmdb.js`. |
| Release parsing / quality detection | Yes | Deterministic regex/string logic; cheap and offline. | Extend `ScraperQualityDetector` from Node `releaseParser.js` features. |
| Scoring / ranking | Yes | Needed for unified dropdown ordering; no network dependencies. | New `ScraperRanker.kt`. |
| Deduplication | Yes | Already exists in addon path via `StreamDeduplicator`; torrent scraper needs same infoHash-first behavior. | Reuse/extend `StreamDeduplicator`. |
| TPB torrent scraping | Yes | Public HTML/API style, returns magnets/info hashes; direct APK OkHttp is feasible. | `providers/torrent/TpbTorrentProvider.kt`. |
| BitSearch torrent scraping | Yes, with caution | Works from server currently; requires HTML parsing and may change markup. | `BitSearchTorrentProvider.kt` + lightweight parser. |
| Nyaa torrent scraping | Yes | Useful anime source; direct HTTP parsing is feasible. | `NyaaTorrentProvider.kt`. |
| 1337x scraping | Partial | Direct HTML often blocks/rate-limits; embed direct attempt but fallback to helper/FlareSolverr. | `LeetxTorrentProvider.kt` with helper fallback. |
| Real-Debrid/AllDebrid/TorBox/Premiumize resolve | Already embedded | Existing debrid clients are better than server proxy. | Keep current `DebridService`. |
| Stremio stream formatting | Yes | App already consumes `Stream` directly; Stremio JSON is only needed for remote addon compatibility. | No Express route needed. |

## What should stay behind an optional helper/backend

| Component | Backend? | Reason |
|---|---:|---|
| FlareSolverr | Yes | Requires headless browser/container; cannot be embedded sanely in Android TV APK. |
| Cloudflare-heavy hosts | Yes | Mobile/TV WebView automation is brittle, slow, and likely blocked. |
| 2Embed / AutoEmbed server extraction | Yes/Hybrid | Inline playback is possible, but extraction often depends on JS challenges, iframe chains, or anti-bot behavior. Prefer helper for extraction; APK only plays final direct/HLS URL. |
| FileMoon / StreamTape / VOE / RabbitStream | Yes | Existing memory says most fail server-side against Cloudflare; embedding them will be worse without a browser automation layer. |
| Torrent cache warming | Optional backend | App can add magnets through debrid, but background batch warming/rate control is better server-side. |
| Remote scraper rule updates | Backend/Worker | APK releases are slow. Use remote JSON provider registry for URLs/selectors/enable flags. |

## Recommended architecture

```text
Unspooled APK
  Content screen
    ↓ imdb/tmdb id
  EmbeddedScraperRepository
    ├─ TorrentSourceProvider[]      // TPB, BitSearch, Nyaa, 1337x direct attempt
    ├─ OptionalHelperProvider       // LAN/helper for FlareSolverr/cloud hosts
    ├─ ScraperRanker
    └─ StreamDeduplicator
        ↓ List<ScraperLink>
  DebridRouter
    ↓ infoHash/directUrl
  DebridService
    ↓ resolved direct URL
  Media3 player
```

## Contract to use inside the APK

Use this as the internal source-provider boundary, not Express/Stremio JSON:

```kotlin
interface EmbeddedTorrentProvider {
    val id: String
    val displayName: String
    val priority: Int
    suspend fun search(context: ScrapeContext): List<TorrentSearchResult>
}

data class ScrapeContext(
    val type: String,
    val imdbId: String?,
    val tmdbId: String?,
    val title: String,
    val year: Int? = null,
    val season: Int? = null,
    val episode: Int? = null,
    val aliases: List<String> = emptyList(),
)

data class TorrentSearchResult(
    val infoHash: String,
    val title: String,
    val source: String,
    val seeders: Int = 0,
    val sizeBytes: Long? = null,
    val magnet: String? = null,
)
```

Then map into existing Android pipeline:

```kotlin
fun TorrentSearchResult.toScraperLink(): ScraperLink = ScraperLink(
    infoHash = infoHash.lowercase(),
    title = title,
    filename = title,
    videoSize = sizeBytes,
    scraperName = source,
)
```

## Port order

1. **Use existing pipeline first**
   - Keep `ScraperProvider`, `StreamScraperEngine`, `ScraperLink`, and `DebridRouter`.
   - Do not add a Node runtime, Express, or local server into the APK.

2. **Add embedded torrent providers**
   - Port `src/providers/torrent/shared.js` helpers to Kotlin.
   - Port TPB first because it is the lowest-risk torrent path.
   - Port BitSearch second using an Android HTML parser.
   - Port Nyaa third.
   - Add 1337x as best-effort direct scrape with helper fallback.

3. **Add a small Android HTML parser dependency**
   - Recommended: `org.jsoup:jsoup`.
   - Do **not** use regex-only HTML parsing for BitSearch/1337x in APK code.

4. **Centralize match/rank/dedupe**
   - Port Node `metadataMatcher.js`, `scoring.js`, and enough of `releaseParser.js`.
   - Dedupe by `infoHash`, then normalized title + size.

5. **Keep helper service optional**
   - Build `HelperScraperClient` pointed at `BuildConfig.NEXSTREAM_SCRAPER_URL` or a new `SCRAPER_HELPER_URL`.
   - Use it only for Cloudflare/browser extraction and fallback providers.

6. **Remote-config source list**
   - Add provider enable flags, base URLs, selectors, timeout, and priority to existing worker/registry path.
   - Keep safe compiled defaults for offline/LAN use.

## Concrete Unspooled changes needed

| Priority | File/Area | Change |
|---:|---|---|
| 1 | `DefaultAddonScrapers.kt` | Stop leaving embedded scraper provider empty. Either populate remote helper config or replace with embedded provider list. |
| 2 | New `data/scraper/embedded/` package | Add `ScrapeContext`, `TorrentSearchResult`, `EmbeddedTorrentProvider`, provider implementations. |
| 3 | `build.gradle.kts` | Add `implementation("org.jsoup:jsoup:<current>")` after checking latest version. |
| 4 | `StreamScraperEngine.kt` | Add second path that runs embedded providers directly instead of only URL-template addon scrapers. |
| 5 | `DebridRouter.kt` | Keep as-is initially; it already accepts `ScraperLink`. Later improve provider selection and avoid slow uncached resolves unless user chooses. |
| 6 | `RealDebridClient.kt` | Keep addMagnet/poll path. Do not reintroduce server `instantAvailability` dependency. |
| 7 | UI source dropdown | Merge embedded torrents + helper/direct sources into one dropdown, sorted by rank/cache/quality. No separated sections. |

## Bad approaches to avoid

| Approach | Why it is wrong |
|---|---|
| Bundle Node.js/Express in the APK | Heavy, fragile, bad Android lifecycle, no TV benefit. |
| Keep `/api/scrape` as the primary app path | Makes the APK dependent on LAN/server availability; contradicts app-embedded goal. |
| Port FlareSolverr/headless browser into APK | Not realistic for Android TV performance or reliability. |
| Regex-only HTML extraction | Already fragile in Node; worse in APK where remote markup changes require app updates. |
| Resolve every uncached torrent automatically | Slow, debrid quota-risky, bad TV UX. Show uncached separately or resolve on selection. |
| Separate “torrent” and “embed” source sections | User explicitly wants one unified dropdown. |

## Immediate implementation target

Build the first APK-native slice like this:

1. `EmbeddedTorrentScraperEngine.search(context)` returns ranked `List<ScraperLink>`.
2. Providers enabled: TPB + BitSearch + Nyaa.
3. 1337x enabled only when direct request succeeds; otherwise helper fallback.
4. Feed links into existing `DebridRouter`.
5. Phone/LAN helper remains optional for browser-protected extraction only.

Acceptance test:

| Test | Expected |
|---|---|
| Movie `tt0133093` | Embedded providers return torrent info hashes without server dependency. |
| No debrid token | App shows sources as available but not instantly playable, or hides debrid-only playback. |
| RD token connected | Cached torrents resolve through `RealDebridClient.getDownloadUrl()` addMagnet/poll path. |
| Helper down | TPB/BitSearch/Nyaa still work; only Cloudflare/embed providers degrade. |
| Phone/TV UI | One unified source dropdown, inline playback only. |