---
name: scraper-debrid-engine-reference
title: Scraper + Debrid Engine Reference
description: Reference for 5 decompiled APKs yielding 637 source files across Streamflix scrapers, STRMR debrid/stremio/trakt, and ONstreamz API contracts. Pure business logic, no UI code.
metadata:
  hermes:
    requires_toolsets: [file]
---

## Source Inventory

Located at `~/apk-analysis/scraper-engine/` — **637 files, 12 MB**, all business logic (no UI).

## Streamflix — Web Scraper Engine (175 Kotlin files)

All pure Kotlin, **zero rewrite needed**. Copy directly.

### Providers — 53 files
- `providers/Provider.kt` — interface all scrapers implement
- `providers/SflixProvider.kt` — English (sflix.to)
- `providers/CineCalidadProvider.kt` — Spanish (cinecalidad.ec)
- `providers/AnimeWorldProvider.kt` — Italian anime (animeworld.ac)
- `providers/StreamingCommunityProvider.kt` — Italian/English (streamingunity.dog)
- `providers/WiflixProvider.kt` — French
- `providers/FilmPalastProvider.kt` — German
- `providers/TmdbProvider.kt` — TMDB metadata enrichment
- ... 45 more

**Pattern:**
```kotlin
object CineCalidadProvider : Provider {
    override val baseUrl = "https://www.cinecalidad.ec"
    override suspend fun getHome(): List<Category> {
        val doc = service.getPage(baseUrl)
        // Jsoup CSS selectors → Movie/TvShow objects
        val featured = doc.select("aside#dtw_content_featured-3 li")
        return listOf(Category("FEATURED", featured))
    }
    override suspend fun getServers(id: String, type: Video.Type): List<Video.Server> {
        val doc = service.getPage(id)
        return doc.select("#playeroptionsul li[data-option]")
            .map { Video.Server(it.attr("data-option"), it.text()) }
    }
    override suspend fun getVideo(server: Video.Server): Video =
        Extractor.extract(server.id, server)
}
```

### Extractors — 82 files
- `extractors/Extractor.kt` — base class + 82-extractor registry + universal bridge
- `extractors/RabbitstreamExtractor.kt` — AES-decrypted sources
- `extractors/StreamtapeExtractor.kt` — direct MP4
- `extractors/DoodLaExtractor.kt` — doodla + doodli variants
- `extractors/VidsrcToExtractor.kt` — embed resolver
- `extractors/FilemoonExtractor.kt` — embed resolver
- ... 77 more

**Pattern:**
```kotlin
suspend fun extract(link: String): Video {
    // 1. Match URL to 1 of 82 extractors by domain
    // 2. Fetch embed page
    // 3. Decrypt AES / unpack JS eval / follow redirects
    // 4. Return Video(source=m3u8, subtitles=[...])
}
```

### Models — 16 files
- `Movie.kt`, `TvShow.kt`, `Episode.kt`, `Season.kt`, `Video.kt`
- `Category.kt`, `Genre.kt`, `People.kt`, `Show.kt`, `WatchItem.kt`
- Per-provider models in `models/animeflv/`, `models/cinecalidad/`, etc.

### Utils — 20 files
- `NetworkClient.kt` — OkHttp factory with Cloudflare bypass headers, DoH, Conscrypt SSL, 4 pre-built clients
- `DecryptHelper.kt`, `CryptoAES.kt`, `AesHelper.kt` — AES decryption for video sources
- `JsUnpacker.kt` — Unpack JavaScript eval() style encryption
- `DnsResolver.kt` — DNS-over-HTTPS via Cloudflare
- `WebViewResolver.kt`, `BypassWebSocket*.kt` — Cloudflare challenge solvers
- `InertiaUtils.kt` — Anti-bot challenge solving
- `TMDb3.kt` — Full TMDB API v3 wrapper

### Jsoup Converter — 4 files
- Custom Retrofit converter factory that parses HTML → Jsoup Document → Kotlin objects

---

## STRMR — Debrid Engine (170 Java→Kotlin files)

### Per-Provider Structure
Each debrid provider has the same layered architecture:

| Layer | Files | What it does |
|---|---|---|
| API service | `data/remote/api/*.java` | Retrofit @GET/@POST interfaces |
| Response models | `data/remote/model/<provider>/*.java` | JSON response schemas |
| Repository | `data/repository/*Repository*.java` | OAuth, magnet, resolve, cache check |
| Entity | `data/local/entity/*.java` | Room entity for token/account storage |
| DAO | `data/local/dao/*Dao*.java` | Room queries |

### Real-Debrid (`debrid/`)
- **API:** `RealDebridApiService.java`, `RealDebridAuthService.java`
- **Models:** DeviceCodeResponse, TokenResponse, CredentialsResponse, AddMagnetResponse, TorrentInfo, TorrentFile, TorrentListItem, InstantAvailability, UnrestrictResponse, UserResponse, DownloadItem, ErrorResponse, CachedFile
- **Repository:** Device code OAuth → poll → exchange → token refresh. Magnet upload → poll → select file → unrestrict. Cache check from torrent list. Download history. Library file resolve.
- **Entity:** `RealDebridAccount` — providerId, accessToken, refreshToken, tokenExpiresAt

### Premiumize (`debrid/`)
- **API:** `PremiumizeApiService.java`, `PremiumizeAuthService.java`
- **Models:** DeviceCodeResponse, TokenResponse, AccountResponse, CacheCheckResponse, DirectLinkResponse, FolderListResponse, Content, Breadcrumb, FolderItem, TransferResponse
- **Repository:** OAuth device code. Direct link resolve. Cache check. Folder browsing + breadcrumbs. Transfers. Library management.
- **Entity:** `PremiumizeAccount`

### AllDebrid (`debrid/`)
- **API:** `AllDebridApiService.java`
- **Models:** PinData, PinCheckData, UserData, MagnetUploadData, MagnetStatusData, MagnetFilesData, MagnetFilesEntry, FileNode, StreamingLink, UnlockData, Quota, Error, LinkFile, AllMagnetsData, UploadedMagnet, MagnetDetail, MagnetError, Response wrapper
- **Repository:** PIN code auth. Magnet upload, status, file selection. Streaming link generation. Quota check. Library browser.
- **Entity:** `AllDebridAccount`

### EasyDebrid (`debrid/`)
- **API:** `EasyDebridApiService.java`
- **Models:** GenerateRequest, GenerateResponse, GeneratedFile, LookupRequest, LookupItem, LookupFile, LookupResponse, UserResponse, ErrorResponse
- **Repository:** Generate (unrestrict) links. Cache lookup by hash. Account info.
- **Entity:** `EasyDebridAccount`

### TorBox (`debrid/`)
- **Entity:** `TorBoxAccount` — schema ready, no API implementation in this APK version

### Shared debrid code (`debrid/`)
- `IDebridProvider.java` — interface all providers implement
- `IDebridLibraryProvider.java` — interface for cloud library browsing
- `DebridProvider.java` — provider metadata enum
- `DebridService.java`, `DebridServiceSection.java` — service grouping
- `DebridFileItem.java` — unified file item model
- `DebridError.java` — standardized error model
- `DebridLibraryRepository.java` — orchestrates all providers' libraries
- `DebridLibrarySearchService.java` — cross-provider search
- `DebridAccountRefreshWorker.java` — WorkManager background token refresh

---

## STRMR — Stremio Addon Protocol (55 Java→Kotlin files)

### Manifest & Catalog Schemas (`stremio/data/model/addon/`)
- `StremioManifest.java` — id, version, name, description, resources, types, catalogs, addonCatalogs, behaviorHints, logo
- `StremioResource.java` — name, types, idPrefixes
- `StremioCatalog.java` — type, id, name, extra[], extraRequired[], extraSupported[]
- `StremioCatalogExtra.java` — name, isRequired, options, optionsLimit
- `StremioManifestBehaviorHints.java` — configurable, configurationRequired
- `AddonConfigField.java` — field key, type, default value
- `DebridInjectionDef.java` — what to inject (resolve debrid streams)
- `DebridSupport.java` — which debrid providers are supported
- `BundleAddonDef.java` — addon bundle definitions (multiple addons in one)

### Protocol Clients (`stremio/data/remote/`)
- `AddonCatalogClient.java` — fetches catalog from addon
- `AddonStreamClient.java` — fetches streams for a title
- `AddonSearchClient.java` — searches addon catalog
- `AddonMetadataClient.java` — fetches metadata/href from addon
- `AddonSubtitleClient.java` — fetches subtitles from addon

### Stream / Subtitle Models (`stremio/data/remote/model/stremio/`)
- `StremioStream.java` — url, behaviorHints, sources, proxyHeaders, title
- `StremioStreamResponse.java` — { streams: [...] }
- `StremioSubtitle.java` — id, url, lang
- `StremioSubtitleService.java` — subtitle API service

### Addon Management
- `AddonRepository.java` — install, uninstall, refresh, installBundle, installConfiguredAddon
- `AddonConfigEntity.java` — persisted addon config
- `InstalledAddonEntity.java` — tracked installed addons
- `AddonConfigDao.java` — Room DAO
- `AddonDisclaimerDao.java` — disclaimer acceptance tracking
- `AddonResolutionService.java` — resolves which addon serves a title
- `StremioAddonService.java` — orchestrates addon resolution
- `StremioSubtitleProvider.java` — provides subs from addon

---

## STRMR — Trakt Integration (118 Java→Kotlin files)

### API Models (`trakt/data/model/trakt/`)
- **Content:** TraktMovie, TraktShow, TraktEpisode, TraktEpisodeFull, TraktIds
- **Progress:** TraktShowProgress, TraktSeasonProgress, TraktEpisodeProgress
- **Sync:** TraktSyncRequest, TraktSyncResponse, TraktSyncCounts, TraktSyncNotFound
- **Scrobble:** TraktScrobbleRequest, TraktScrobbleResponse, TraktScrobbleMovie, TraktScrobbleEpisode, TraktScrobbleShow, TraktScrobbleIds, TraktScrobbleSharing
- **History:** TraktHistoryItem, TraktWatchedMovie, TraktWatchedShow, TraktWatchedSeason, TraktWatchedEpisode
- **Watchlist:** TraktWatchlistItem
- **Collection:** TraktCollectionItem
- **Ratings:** TraktRatedMovie, TraktRatingItem, TraktRatingRequest
- **Lists:** TraktMediaList, TraktListItem, TraktListIds, TraktLikedList, TraktPublicList, TraktUserList, TraktCreateListRequest
- **Auth:** TraktDeviceCodeResponse, TraktTokenResponse, TraktUser, TraktUserIds, TraktUserStats, TraktUserMovieStats, TraktUserShowStats
- **Search:** TraktSearchResult, TraktSearchEpisodeResult
- **Misc:** TraktCalendarEntry, TraktActivityGroup, TraktHiddenItem, TraktPlaybackItem, TraktLastActivities, TraktPaginationInfo, TraktTrendingMovie, TraktTrendingShow

### Repositories
- `TraktAuthRepository.java` — device code OAuth, token management
- `TraktScrobbleRepository.java` — startWatching, stopWatching, updateProgress
- `TraktSyncRepository.java` — sync collection, history, watchlist (incremental + full)
- `TraktMediaRepository.java` — fetch lists, pagination, content building
- `TraktAccountRepository.java` — link/unlink Trakt accounts
- `TraktOfflineQueueRepository.java` — offline action queue (scrobbles that failed)
- `TraktStatusProvider.java` — Trakt connection status
- `TraktActivitiesCache.java` — cache last activities for efficient polling
- `EpisodeRatingsRepository.java` — fetch episode ratings

### Background Workers
- `TraktSyncWorker.java` — WorkManager-based periodic sync
- `TraktOfflineQueueWorker.java` — retry failed offline actions

### ETag Caching
- `TraktETagInterceptor.java` — OkHttp interceptor that sends ETag/If-None-Match for bandwidth-efficient polling

---

## STRMR — Auth & Infrastructure (43 Java→Kotlin files)

### Auth (`strmr/auth/`)
- `GoogleAuthManager.java`, `GoogleUser.java` — Google sign-in
- `FeatureGateManager.java`, `Feature.java`, `GateResult.java`, `BlockReason.java` — feature gating based on auth state
- `UserAccountEntity.java` — Room entity for user accounts

### Player Entities (`strmr/player/`)
- `PlaybackProgress.java` — resume position, content ID, progress ms
- `PlayerSettings.java` — per-user player config (speed, subtitles, quality)
- `EpisodeWatchStatus.java` — watched/unwatched per episode
- `PlaybackProgressDao.java`, `PlayerSettingsDao.java`, `EpisodeWatchStatusDao.java` — Room DAOs

### Android TV Channels (`strmr/channels/`)
- `HomeChannelManager.java` — create/update/delete TV home screen channels
- `HomeChannelUpdateWorker.java` — WorkManager for periodic channel refresh
- `ChannelImageProvider.java` — serves channel logo images
- `CatalogRefreshWorker.java` — refreshes channel catalog content

### DB Entities (`strmr/models/`)
- `ContinueWatchingEntity.java` — resume playback tracking
- `CachedContent.java` — offline cached content metadata
- `WatchStatusEntity.java` — watch progress per item
- `SyncCheckpoint.java` — last sync timestamps
- `SyncMetadataEntity.java` — sync state metadata
- `UserCollectionEntity.java` — user's saved library
- `MediaEnrichmentEntity.java` — enriched media metadata from TMDB/Trakt
- `ContentImageOverrideEntity.java` — custom poster/backdrop overrides
- Plus 7 DAO files with queries
- `RowConfigEntity.java`, `RowConfigDao.java` — home screen row configuration

---

## ONstreamz — Backend API Contract (76 Java→Kotlin files)

### Request Models (`onstreamz/api/request/`)
- `LoginRequest`, `LoginCodeRequest`, `RegisterRequest`, `RefreshTokenRequest`
- `LoginWithGoogleRequest`, `ForgotPasswordRequest`, `ChangePasswordRequest`
- `SendVerifyEmailRequest`, `UpdateUserInfoRequest`, `LogoutRequest`
- `AddToWatchListRequest`, `ContinueWatchRequest`, `RatingRequest`, `ReportRequest`
- `CreateTurnSearchRequest`

### Response Models (`onstreamz/api/response/`)
- `HomeResponse` — sections, rows, items, banners
- `MovieResponse`, `MovieListResponse`, `EpisodeResponse`, `EpisodeDetailResponse`
- `SeasonResponse`, `ListSeasonResponse`, `ListEpisodeResponse`
- `GenreResponse`, `GenreListResponse`, `CastResponse`, `CompanyResponse`
- `StreamDataResponse`, `StreamingResponse`, `PlayerResponse`, `ListPlayerResponse`
- `SubResponse` — subtitles
- `LoginResponse`, `TokenResponse`, `TokenDataResponse`, `RefreshTokenResponse`
- `UserInfoResponse`, `UserResponse`
- `AllSettingsResponse`, `VersionSettingsResponse`, `LatestVersionResponse`
- `FilterResponse`, `PageDataResponse`, `ResultResponse`
- `ContinueWatchResponse`, `InWatchlistResponse`
- `DownloaderResponse`, `FileSizeResponse`
- `ErrorResponse`, `VoteResponse`
- `AvatarResponse`, `ListAvatarsResponse`
- `ContactLinkResponse`, `ShortcutResponse`, `TabResponse`
- `TopicReportResponse`, `ListTopicReportResponse`, `CountryResponse`

### Domain Models (`onstreamz/models/`)
- `Movie.java`, `Episode.java`, `Season.java`, `Genre.java`, `Cast.java`, `Company.java`
- `Country.java`, `Avatar.java`, `Filter.java`, `Vote.java`
- `Shortcut.java`, `LatestVersion.java`
- `StreamSource.java`, `StreamUrl.java`, `SubtitleSource.java`, `EpisodeSource.java`
- `MediaType.java` — enum (MOVIE, TV_SHOW)
- `GeneralPreference.java`

---

## Critical Flow: Site-Specific IDs

The #1 mistake when porting Android scrapers: **use site-internal IDs, not TMDB IDs.**

Native Android APKs (Streamflix, etc.) work because:
1. User searches WITHIN the site (e.g., Sflix search) → gets site-specific page URLs
2. Those URLs contain the site's own content IDs
3. IDs flow through: search → detail → servers → play

When porting, replicate this flow:
```
TMDB title → search SITE for title → extract SITE's page URL/ID → get servers using site ID
```
Passing raw TMDB IDs directly to site providers returns 404s or homepage HTML.

## Cloudflare Bypass (2026)

Server-side Cloudflare bypass requires a real browser engine. uTLS/Go HTTP alone WILL be blocked.

| Tool | Lang | Pass Rate | Notes |
|------|------|-----------|-------|
| **nodriver** (ultrafunkamsterdam) | Python | 28/31, 0 blocked | Best. `pip install nodriver`. Direct CDP, no WebDriver. async. Use `browser.get(url)` then `tab.evaluate("JS")` to extract content. Close with `await browser.aclose()`. |
| FlareSolverr 3.5 | Python | Good | Docker sidecar, HTTP API, most maintained |
| chromedp | Go | OK | Needs `--no-sandbox`. Verify `chromium` binary path. 3-8s/page. |

**nodriver is the best 2026 option** — successor to undetected-chromedriver, direct Chrome DevTools Protocol, no Playwright middleware.

### nodriver Quick Pattern
```python
import nodriver as nd, asyncio, json

async def fetch(url: str) -> str:
    b = await nd.start(headless=True, sandbox=False, disable_gpu=True)
    tab = await b.get(url)
    await asyncio.sleep(5)  # Wait for Cloudflare challenge
    html = await tab.evaluate("document.documentElement.outerHTML")
    await b.aclose()
    return html

result = asyncio.run(fetch("https://site.com"))
```
Key flags: `headless=True, sandbox=False, disable_gpu=True`. The `evaluate()` method runs JavaScript in the page context and returns the result. Use `evaluate` for HTML content, NOT `tab.content()` (doesn't exist in nodriver API).

## Real-Debrid API: Cache Checking (2026 Update)

**`/torrents/instantAvailability/{hash}` is DEPRECATED** — returns `{"error": "disabled_endpoint", "error_code": 37}`.

### Correct approach: addMagnet + check status
```javascript
// 1. Add magnet
const addResp = await axios.post(`${RD_API}/torrents/addMagnet`,
  `magnet=${encodeURIComponent(magnetLink)}`,
  { headers: rdHeaders() }
);
const torrentId = addResp.data.id;

// 2. Check info to see if instantly cached
const infoResp = await axios.get(`${RD_API}/torrents/info/${torrentId}`, { headers: rdHeaders() });
const isCached = infoResp.data.status === 'waiting_files_selection' || infoResp.data.status === 'downloaded';

// 3. Clean up (delete the check torrent)
await axios.post(`${RD_API}/torrents/delete/${torrentId}`, null, { headers: rdHeaders() });
```
Magnet format: `magnet:?xt=urn:btih:HASH&dn=video&tr=udp://tracker.opentrackr.org:1337`

### Multi-User Debrid Pattern
For apps where each user has their own RD key (Stremio-style), pass the key per-request:
```
GET /stream/movie/:id.json?rdKey=USER_API_KEY
```
The scraper uses the user's key (not a server-configured key) for all RD operations. The `rdHeaders(userKey)` function falls back to `config.RD_API_KEY` when no user key is provided.

## Browser Playback: MKV Transmuxing

Real-Debrid returns MKV files (especially REMUX). Browsers CANNOT play MKV natively. The fix: transmux MKV → fragmented MP4 via ffmpeg (copies video/audio, no re-encode).

```javascript
const ffmpeg = spawn('ffmpeg', [
  '-i', rdDirectUrl,
  '-c', 'copy',                              // Copy codecs, no re-encode
  '-movflags', 'frag_keyframe+empty_moov',   // Fragmented MP4 for instant playback
  '-f', 'mp4',
  'pipe:1',
]);
ffmpeg.stdout.pipe(res);  // Stream to browser
```
Available via Alpine: `apk add --no-cache ffmpeg`. Near-zero CPU since `-c copy` doesn't re-encode.

## Embed Sources Are Free

Embed sites (Vidsrc, Filemoon, Voe, Streamtape, Rabbitstream) are **free** — no debrid subscription. Quality is typically 720p-1080p. The tradeoff vs torrents+RD:

| | Embed Sources | Torrent + RD |
|---|---|---|
| Cost | Free | $3-4/mo |
| Quality | 720p-1080p | Up to 4K REMUX |
| Reliability | Sites change/block | Stable CDN |
| Setup | Just works | Need RD key |

### Top Embed Hosters (from Streamflix extractors)
| Hoster | Technique | Complexity |
|--------|-----------|------------|
| Filemoon | ECDSA secp256r1 attestation + AES-256-GCM decrypt | **High** — generates EC keypair, signs challenge, submits JWK |
| Rabbitstream/Megacloud | AES-CBC decrypt from player script | Medium |
| VidsrcNet | 10 different decryption functions (ROT13, Caesar, hex-xor, base64+shift) | Medium |
| VidsrcTo | RC4 token encoding | Medium |
| Streamtape | Regex extraction from innerHTML | Low |
| Voe | Redirect chain + iframe | Low |

- **Java files need Kotlin translation:** All STRMR and ONstreamz files are Java. Android Studio auto-converts paste, or use Java→Kotlin converter. The interfaces and models are straightforward; the repositories need coroutine adapters.
- **Hilt/Dagger injected repos:** STRMR repos use `@Inject` + `*_Factory.java` files. Strip the Dagger boilerplate, inject manually or use your own DI.
- **Room _Impl files are auto-generated:** The `*Dao_Impl.java` files are Room-generated. You only need the `*Dao.java` interface + `*Entity.java` + `AppDatabase.java` for schema reference.
- **Moshi adapters:** ONstreamz uses Moshi. The `*JsonAdapter.java` files are code-gen. You only need the raw model classes for JSON schema reference.
- **Real-Debrid API rate limits:** The realdebrid repos handle this but you may need to adapt retry logic.
- **Stremio addon URLs need a resolver:** The App is hardcoded to use the core addons from the official Stremio server. You'll need to provide your own addon catalog or use the Stremio community addons endpoint.
- **Trakt API requires client ID:** All Trakt requests pass an OAuth client ID. You'll need your own from trakt.tv/oauth/applications.
- **Some endpoints may be stale:** The STRMR APK is v1.1.72 — API schemas may have minor differences from current debrid provider APIs.