--- name: kotlin-to-go-porting title: Kotlin to Go Code Porting (Android Scrapers → Go Servers) description: Port Kotlin/Android scraper and extractor code to Go server code — converting Jsoup HTML parsing, Android-specific crypto, Retrofit HTTP, and WebView-dependent logic into idiomatic Go equivalents. domain: software-development metadata: hermes: requires_toolsets: [terminal, file] --- # Kotlin → Go Porting for Android Scraper Code How to systematically port Android scraper code (Kotlin, Jsoup, Retrofit, OkHttp) to a Go server backend. Covers the patterns used to port 83 extractors + 9 site providers from the Streamflix APK scraper engine to the ultimate-scraper Go server. ## Architecture Translation | Kotlin/Android Pattern | Go Equivalent | |---|---| | `Jsoup.select("css.selector")` | `regexp.FindStringSubmatch` / `strings` package | | `JsoupConverterFactory` (Retrofit → HTML objects) | Manual `regexp` parsing + struct construction | | `Retrofit` API interfaces | `net/http` GET/POST with `http.Client` | | `OkHttpClient` with custom DNS | `http.Transport` with `DialContext` | | `suspend fun extract(...)` | `func Extract(...) (*T, error)` | | `object SingletonProvider : Provider` | `struct` + `init()` registration | | `android.util.Base64` | `encoding/base64` | | `javax.crypto.Cipher` | `crypto/aes`, `crypto/cipher`, `crypto/rc4` | | `WebView` + `suspendCancellableCoroutine` | `http.Client` with redirect-following or headless browser | | `kotlinx.coroutines` | `sync.WaitGroup` + goroutines | | `Gson` / `GsonBuilder` | `encoding/json` | ## Provider Interface Pattern ```kotlin // Kotlin (Streamflix) interface Provider { val baseUrl: String val name: String suspend fun getHome(): List suspend fun search(query: String, page: Int = 1): List suspend fun getMovie(id: String): Movie suspend fun getServers(id: String): List suspend fun getLink(serverId: String): String } ``` ```go // Go (ultimate-scraper) type SiteProvider interface { Name() string BaseURL() string Language() string GetHome() ([]models.Category, error) Search(query string, page int) ([]models.AppAdapterItem, error) GetMovie(id string) (*models.Movie, error) GetTvShow(id string) (*models.TvShow, error) GetServers(id string) ([]models.Server, error) } func RegisterProvider(p SiteProvider) // called in init() func GetAllProviders() []SiteProvider // iterate all func GetProviderByName(name string) SiteProvider ``` ## Extractor Interface Pattern ```kotlin // Kotlin (Streamflix) abstract class Extractor { abstract val name: String abstract val mainUrl: String open val aliasUrls: List = emptyList() abstract suspend fun extract(link: String): Video } ``` ```go // Go type Extractor interface { Name() string MainURL() string Extract(link string) (*models.Video, error) } func Register(e Extractor) // called in init() func Resolve(embedURL string) (*models.Video, error) ``` ## HTML Parsing Conversion Jsoup CSS selectors → Go regexp patterns: | Jsoup | Go regexp | |-------|-----------| | `doc.select("div.swiper-slide")` | `(?s)]*class="[^"]*swiper-slide[^"]*"(.*?)` | | `element.attr("href")` | capture group for `href="([^"]+)"` | | `element.text()` | `(?s)>(.*?)<` (strip HTML tags) | | `element.selectFirst("img.film-poster-img")` | `]*class="[^"]*film-poster-img[^"]*"` | | `elements.toInfo()` | manual field-by-field regex | Common helper: `fetchPage(url string) ([]byte, error)` wraps `http.Client` GET with timeout and UA header. ## Crypto Porting ### AES/CBC/PKCS5 (Rabbitstream, Chillx) ```go // Go: crypto/aes + crypto/cipher func DecryptAESCBC(data []byte, key, iv []byte) ([]byte, error) ``` - Kotlin `Cipher.getInstance("AES/CBC/PKCS5Padding")` → Go `cipher.NewCBCDecrypter` - MD5 KDF: call `md5.Sum()` iteratively (OpenSSL EVP_BytesToKey style) ### RC4 (Vidsrc.to, Vidplay) ```go // Go: crypto/rc4 func DecryptRC4(data, key []byte) ([]byte, error) ``` - Kotlin KSA/PRGA initialization → Go `rc4.NewCipher` - XOR keystream → Go `Cipher.XORKeyStream` ### AES/GCM (Filemoon) ```go // Go: crypto/aes + crypto/cipher func DecryptAESGCM(ciphertext, key, nonce []byte) ([]byte, error) ``` - Kotlin `GCMParameterSpec` + `Cipher.getInstance("AES/GCM/NoPadding")` → Go `cipher.NewGCM` ### JsUnpacker (P.A.C.K.E.R.) Regex match `eval(function(p,a,c,k,e,` pattern → extract payload, radix, count, keywords → unpack via dictionary substitution. ## WebView Workarounds Kotlin uses `WebView` for: - Redirect-following (StreamWish, Upzone) - JavaScript evaluation - Cookie collection - Cloudflare challenge solving Go alternatives: - `http.Client` with `CheckRedirect` for simple redirect chains - Regex-based extraction from HTML + JS for simple cases - For complex cases: log as "simplified" and try direct m3u8 regex - **For Cloudflare bypass**: use `chromedp` with headless Chromium ### Headless Chromium for Cloudflare Bypass When all else fails (sites behind Cloudflare JS challenge), install headless Chromium and use chromedp: ```bash sudo apt-get install -y chromium go get github.com/chromedp/chromedp@latest ``` Basic pattern (see `scraperlib/browser.go` in ultimate-scraper): ```go import "github.com/chromedp/chromedp" func BrowserResolve(embedURL string) (string, error) { allocCtx, cancel := chromedp.NewExecAllocator(context.Background(), chromedp.Headless, chromedp.DisableGPU, chromedp.UserAgent("Mozilla/5.0 ... Chrome/124.0.0.0 Safari/537.36"), ) defer cancel() ctx, cancel := chromedp.NewContext(allocCtx) defer cancel() ctx, cancel = context.WithTimeout(ctx, 30*time.Second) defer cancel() var m3u8 string err := chromedp.Run(ctx, chromedp.Navigate(embedURL), chromedp.Sleep(3*time.Second), // wait for CF challenge to pass chromedp.EvaluateAsDevTools(`... JS to find m3u8 ...`, &m3u8), ) return m3u8, err } ``` Caveats: - Adds ~200MB dependency (Chromium binary) - Each resolve takes 3-8 seconds (launch browser, wait CF, extract) - Production: pool browser instances or use a browserless service ## Parallel Porting with Subagents For large batches (50+ files), use `delegate_task` with up to 3 parallel subagents. This pattern was proven at scale porting 83 extractors + 9 providers. ### Splitting Strategy Split work by **complexity**, not by module: | Batch | What | Method | |---|---|---| | Complex (4-8 files) | Rabbitstream, Megacloud, Vidsrc.to, Vidplay, VixSrc, Chillx, Filemoon | Each subagent reads the exact Kotlin source, replicates full crypto/API logic in Go | | Simple (50-60+ files) | All regex-based hosters (Vidoza, BigWarp, LoadX, Goodstream, etc.) | **Template pattern**: each follows `GET page → regex m3u8 → if packed JS, unpack → return Video` | | Providers (3-9 files) | Site scrapers (Sflix, CineCalidad, TMDB, Cuevana, etc.) | Full HTML parsing with regex, follows the SiteProvider interface | ### Context Requirements Every subagent MUST receive: 1. The **existing Go interface definitions** (Extractor interface, SiteProvider interface, registry functions) 2. The **existing crypto package** (AES-CBC, RC4, JsUnpacker, Base64 variants, M3U8Regex, MP4Regex) 3. The **models package** (Video, Subtitle, Movie, TvShow, Category, Server structs) 4. The exact **Kotlin source file path** to read — DO NOT inline the Kotlin code, let the subagent read it 5. A clear **output file path** and naming convention ### Simple Extractor Template For simple regex-based hosters, include this exact pattern in the subagent context: ```go type XExtractor struct{} func (e *XExtractor) Name() string { return "X" } func (e *XExtractor) MainURL() string { return "domain.com" } func (e *XExtractor) Extract(link string) (*models.Video, error) { body, err := fetchString(link) if err != nil { return nil, err } if m := crypto.M3U8Regex.FindString(body); m != "" { return MakeVideo(m, nil), nil } result := crypto.UnpackJs(body) if result.Success { if m := crypto.M3U8Regex.FindString(result.Data); m != "" { return MakeVideo(m, nil), nil } } return nil, fmt.Errorf("no source found in %s", link) } func init() { Register(&XExtractor{}) } ``` ### Post-Merge Steps ```bash go build ./... # catch compile errors (type mismatches, missing imports) grep 'Register(' *.go # check for duplicate registrations (last wins silently) grep 'init()' *.go # verify all init() functions exist ``` ### Failure Handling - If a subagent fails due to API auth error (e.g. `401 Authentication Fails`): re-dispatch the task manually from the parent session — the subagent's model credential isn't configured for that run's provider - If a subagent produces duplicate registrations: the later `init()` silently overwrites the earlier one. Deduplicate by removing the duplicate from `simple.go` or `common.go` - If build fails: check for missing imports (`crypto`, `models`, `fmt`), wrong function signatures, or type mismatches between the new files and existing code ## The One-Click Play Pattern (Critical) The Streamflix APK user experience: **browse → tap movie → tap play → video plays**. No source selection, no "Find Sources" button, no choosing between embed URLs. This is the UX to replicate. ### Original APK Flow ``` 1. User browses (provider.getHome()) → gets SITE-SPECIFIC IDs 2. User taps movie → MovieViewModel(id) → provider.getMovie(id) ← SITE ID 3. Movie detail shows with PLAY button 4. User taps PLAY → PlayerViewModel(id) → provider.getServers(id) ← SITE ID 5. → provider.getVideo(server) → Extractor.extract(embedURL) ← RESOLVE 6. → m3u8 → plays in ExoPlayer ``` **ID CONSTRAINT**: The IDs flow from the provider's own search/catalog. They are site-specific slugs or page URLs (e.g., `/watch-movie-27205`), NOT TMDB IDs (e.g., `27205`). This is the single most critical difference between a working and non-working port. ### Correct Implementation ```go // WRONG — passes TMDB ID directly, site doesn't recognize it servers, err := provider.GetServers("27205") // TMDB ID for Inception // RIGHT — first search the provider for the movie title results, err := provider.Search("Inception", 1) siteID := results[0].ID // e.g., "/watch-movie-27205" servers, err := provider.GetServers(siteID) ``` ### How the APK Handles This The APK uses the SELECTED PROVIDER for ALL content discovery: 1. Home screen shows the provider's catalog (via `getHome()`) 2. Search uses the provider's search (via `search(query, page)`) 3. Movie detail uses the provider's detail page (via `getMovie(id)`) 4. Play uses the provider's server list (via `getServers(id)`) The IDs are ALWAYS the provider's internal format. TMDB is only used for enrichment (posters, ratings, cast) via a separate TMDB provider, never for the playback flow. ### Frontend One-Click Architecture When user taps a movie card: ```javascript async function onMovieClick(movieId, type) { // 1. Show detail immediately from TMDB (fast, always works) const tmdbData = await api(`/api/tmdb/${type}/${movieId}`); showDetail(tmdbData); // 2. In background: find playable source (the slow part) const links = await api(`/links/movie/${movieId}`); const embedUrl = findBestSource(links); // 3. When user taps PLAY, start embed immediately // (no source selection screen) playEmbed(embedUrl); } ``` ## Cloudflare Bypass Strategy ### Reality: Server-Side Cloudflare Bypass is Extremely Limited uTLS (`refraction-networking/utls`) with any preset (`HelloChrome_Auto`, `HelloChrome_120`) will NOT bypass Cloudflare JS challenges. The `"malformed HTTP response"` error (bytes like `\x00\x00\x12\x04...`) is Cloudflare's binary challenge page — a non-HTTP response sent when the TLS fingerprint doesn't match a real browser. **Why the APK worked**: Android's WebView is a full browser engine that: - Executes JavaScript challenge code - Sets proper cookies from challenge responses - Has a real TLS stack (Conscrypt) - Presents a real browser fingerprint at the TCP/TLS level ### Working Server-Side Approach: chromedp ```bash sudo apt-get install -y chromium go get github.com/chromedp/chromedp@latest ``` ```go import "github.com/chromedp/chromedp" func browserResolve(embedURL string) (string, error) { allocCtx, _ := chromedp.NewExecAllocator(context.Background(), append([]chromedp.ExecAllocatorOption{ chromedp.NoSandbox, // REQUIRED for non-root/container chromedp.Headless, chromedp.DisableGPU, chromedp.Flag("disable-software-rasterizer", true), chromedp.UserAgent("Mozilla/5.0 ... Chrome/124..."), }, chromedp.DefaultExecAllocatorOptions[:]...)..., ) ctx, _ := chromedp.NewContext(allocCtx) ctx, cancel := context.WithTimeout(ctx, 8*time.Second) defer cancel() var sources string chromedp.Run(ctx, chromedp.Navigate(embedURL), chromedp.WaitReady("body"), chromedp.Sleep(3*time.Second), // wait for CF challenge to pass chromedp.EvaluateAsDevTools(`... JS to find m3u8 ...`, &sources), ) return sources, nil } ``` Key configuration requirements: - **`chromedp.NoSandbox`** — Required when running as root or inside Docker containers - **`chromedp.DisableGPU` + `disable-software-rasterizer`** — Prevents GPU-related hangs in headless mode - **3-5 second initial sleep** — Cloudflare challenges take 1-3 seconds to auto-solve - Timeout: 8 seconds is usually enough for CF bypass + page load - Chromium will auto-detect the binary path — no need to configure `ExecPath` ### Alternative: nodriver (Python, Best in 2026) ⭐ PRIMARY RECOMMENDATION [ultrafunkamsterdam/nodriver](https://github.com/ultrafunkamsterdam/nodriver) — successor to undetected-chromedriver. According to a 2026 anti-detect browser benchmark (31 Cloudflare targets, 3 runs each), nodriver scored **28/31 targets passed, 0 blocked** — the best result of all tested tools. It communicates directly with Chrome DevTools Protocol (no Selenium/Playwright middleware), making it harder to detect. Install: ```bash pip install nodriver ``` Basic pattern: ```python import nodriver as nd async def fetch(url): browser = await nd.start(headless=True, sandbox=False, disable_gpu=True) tab = await browser.get(url) await tab.wait_for("body", timeout=15) await asyncio.sleep(3) html = await tab.evaluate("document.documentElement.outerHTML") await browser.aclose() # NOT `close()` — nodriver uses aclose return html ``` Call from Go via subprocess: ```go cmd := exec.Command(venvPython, script, url, fmt.Sprintf("%.0f", timeout.Seconds())) output, err := cmd.Output() // output is JSON with {"success": true, "html": "...", "html_length": N} ``` Two modes for the Python script: - `html` mode: returns full page HTML after Cloudflare bypass - `resolve` mode: returns extracted video sources (m3u8, mp4, iframe URLs) Key API differences from chromedp: - `await browser.start()` not `chromedp.NewContext()` - `await browser.get(url)` returns a tab, not `chromedp.Run()` - `tab.evaluate(js)` not `chromedp.EvaluateAsDevTools()` - `await browser.aclose()` not `browser.Close()` - No separate chromedriver binary needed ### Fallback: chromedp (Go, heavier) If nodriver is not available, use chromedp with headless Chromium. Requires `chromedp.NoSandbox` flag when running as root or in containers. ### Alternative: FlareSolverr (Docker) ```bash docker run -d --name=flaresolverr -p 8191:8191 flaresolverr/flaresolverr:latest curl -X POST http://localhost:8191/v1 -d '{"cmd": "request.get", "url": "..."}' ``` Battle-tested, Python/Docker based. ## Real-Debrid Integration When returning torrent streams (info hashes) from scrapers, integrate with Real-Debrid for instant playback of cached content. ### Cache Checking The `instantAvailability` endpoint is **deprecated** by RD (error code 37: `disabled_endpoint`). Instead, check cache by adding the magnet and inspecting the response status: ```javascript // Add magnet to RD const magnetLink = `magnet:?xt=urn:btih:${HASH}&dn=video&tr=udp://tracker.opentrackr.org:1337`; const addResp = await fetch('/rd-post/torrents/addMagnet', { method: 'POST', headers: { 'Content-Type': 'application/x-www-form-urlencoded' }, body: `magnet=${encodeURIComponent(magnetLink)}` }); const addData = await addResp.json(); const rdId = addData.id; // Check if cached const infoResp = await fetch(`/rd/torrents/info/${rdId}`); const info = await infoResp.json(); const isCached = info.status === 'waiting_files_selection' || info.status === 'downloaded'; // Clean up check torrent await fetch(`/rd-post/torrents/delete/${rdId}`, { method: 'POST' }); ``` ### Playback Flow ```javascript // 1. Add magnet const addResp = await fetch('/rd-post/torrents/addMagnet', { ... }); const torrentId = addData.id; // 2. Select files await fetch(`/rd-post/torrents/selectFiles/${torrentId}`, { ... }); // 3. Wait for RD to process (poll status) let info; for (let i = 0; i < 15; i++) { await new Promise(r => setTimeout(r, 1500)); info = await fetch(`/rd/torrents/info/${torrentId}`).json(); if (info.status === 'downloaded') break; } // 4. Get the download link const link = info.links[0]; // 5. Unrestrict to get playable URL const urResp = await fetch('/rd-post/unrestrict/link', { method: 'POST', body: `link=${encodeURIComponent(link)}` }); const playUrl = (await urResp.json()).download; ``` ### Browser Playback Limitation RD returns MKV files for BluRay REMUX sources. **Browsers cannot play MKV natively.** Solutions: | Solution | Pros | Cons | |---|---|---| | Direct RD link → VLC/MPV | Always works, full quality | External player required | | RD transcode endpoint | Browser-compatible HLS | Deprecated by RD (unknown_method) | | ffmpeg server-side transcode | Any format → HLS | CPU intensive, complex | | Show direct link for VLC | Simple, reliable | User needs VLC installed | Best practice: provide a "Open in VLC" button with the direct RD link, plus a copy-link button for power users. ### CORS Proxy RD API does **not** return CORS headers. All RD API calls from the browser must go through a server-side proxy. Pattern: ```python # Python proxy server class ProxyHandler(http.server.SimpleHTTPRequestHandler): def do_GET(self): if self.path.startswith('/rd/'): path = self.path[4:] url = f'https://api.real-debrid.com/rest/1.0/{path}' req = urllib.request.Request(url) req.add_header('Authorization', f'Bearer {RD_KEY}') # ... proxy and return with CORS headers ``` Protect the RD API key by keeping it server-side only — never expose it in browser JavaScript. ### Reference files - `references/ultimate-scraper-port.md` — Full port details (83 extractors, 9 providers, crypto, deployment) - `references/streaming-site-arch.md` — Streaming site architecture, Cloudflare lessons, TMDB proxy, embed vs direct playback strategy - `references/scraper-tools-inventory.md` — Scraper repos on CachyOS (FlareSolverr, nodriver, vidsrc-bypass, yt-dlp, etc.) - **TMDB proxy must reconstruct query strings**: Fiber's `c.Params("*")` strips query parameters. Always use `c.Request().URI().QueryString()` to rebuild: `fmt.Sprintf("https://api.themoviedb.org/3/%s?%s&api_key=%s", path, string(q), apiKey)`. If query string is empty, use `?api_key=...`. - **Duplicate JS function declarations break frontend silently**: When patching HTML files, a duplicate `async function findSources()` on consecutive lines causes JavaScript to halt silently. The `loadHome()` call never executes, so the catalog never loads. Verify with `grep -c '^async function' index.html` after edits — any count > 1 implies duplicates. - **Duplicate registrations**: Multiple `init()` functions registering the same extractor name — Go doesn't complain, last wins silently. Check with `grep 'Register(' *.go` after merge. - **WebView-dependent extractors**: Many Kotlin extractors use `WebView` for redirect resolution. Go has no built-in WebView. Fall back to regex extraction or HTTP redirect following. For complex cases (Filemoon ECDSA attestation, StreamWish redirect chains), implement the full HTTP flow with `net/http` and `crypto/ecdsa` — Go can replicate the logic without a browser. - **Android `Base64.URL_SAFE`**: Different from Go's `base64.URLEncoding`. Use `base64.RawURLEncoding` and manually add padding. - **Jsoup's `selectFirst`**: May return null; Kotlin uses `?: throw`. Go has no null safety — check error returns explicitly. - **Retrofit's `@Url`**: Dynamic URLs passed at call time. Go equivalent: construct full URL before the request. - **Cipher parameters**: Kotlin's `cipher.parameters` returns `AlgorithmParameters` that can be serialized. Go needs manual IV extraction. - **Host key mismatch for SSH to CachyOS**: SSH config uses user `raymond` (not `rurouni`). Use `ssh cachyos` (host alias in ~/.ssh/config). - **Fiber `app.Static()` is shallow**: `app.Static("/assets", "./frontend/assets")` only serves files under that prefix. For a standalone test page like `frontend/test.html`, add an explicit route: `app.Get("/test", func(c *fiber.Ctx) error { ... })` that reads the file and sends it. - **Provider registration via `init()` is automatic**: Providers call `RegisterProvider()` in their `init()` functions. No manual wiring in `main()` is needed. Just import the providers package and they auto-register. - **Duplicate provider/extractor in multiple files**: After merging files from parallel subagents, an extractor may exist in both `common.go` (older ported version) and `simple.go` (new batch port). Check with `grep -c 'Register.*Name' *.go | grep -v ':1$'` to find duplicates. The last `init()` wins — keep the more complete implementation. - **Server-side Cloudflare bypass is unreliable**: uTLS (`refraction-networking/utls`) with `HelloChrome_Auto` or `HelloChrome_120` will NOT bypass Cloudflare JS challenges. The "malformed HTTP response" (bytes like `\\x00\\x00\\x12\\x04...`) is Cloudflare's binary challenge page. This is a fundamental limitation: Go servers cannot execute JavaScript. Sites protected by Cloudflare Turnstile/challenge can only be scraped from environments with a full JS runtime (Android WebView, headless Chrome, user's browser). - **When embed URLs can't be resolved server-side, use iframe fallback**: The frontend should IMMEDIATELY load embed URLs as iframes when clicked. The user's browser handles Cloudflare/JS natively. Direct m3u8 resolution via server-side extractors is a SECONDARY optimization that runs in background. Architecture: `click -> show iframe (instant) -> try resolve in background -> if resolved, show "Switch to Direct" button`. - **TMDB proxy must reconstruct query strings**: Fiber's `c.Params("*")` strips query parameters. Always use `c.Request().URI().QueryString()` to rebuild: `fmt.Sprintf("https://api.themoviedb.org/3/%s?%s&api_key=%s", path, string(q), apiKey)`. If query string is empty, use `?api_key=...`. - **handleLinks must try ALL registered providers**: Iterate `providers.GetAllProviders()` and dispatch each provider's `GetServers()` in a goroutine with `sync.WaitGroup`. Skip TMDB (catalog-only). Each provider gets a 15s timeout. Deduplicate by URL. Sort HLS (resolved) sources before webpage sources. - **Site scrapers need site-specific IDs**: Movie sites use their own URL slugs (`inception` not `27205`). `GetServers()` fails with raw TMDB IDs. Correct flow: use provider's `Search()` to find site-local ID first, then pass that to `GetServers()`. - **Embed aggregator URLs (VidSrc family) are unreliable**: Hardcoded URLs like `vidsrc.icu`, `autoembed.co`, `multiembed.mov` go dead frequently. They are fallback, not primary. Site providers (CineCalidad, Cuevana) find embed URLs from working sites, but those sites are Cloudflare-protected. - **Fiber endpoints should return 200+JSON on failure**: Never return HTTP 502 on resolution failure. Always return 200 + `{"success": false, "error": "...", "note": "...", "url": "..."}`. Let the frontend decide how to handle the failure. - **Duplicate JS function declarations break frontend silently**: When patching HTML files, a duplicate `async function findSources()` on consecutive lines causes JavaScript to halt silently. The `loadHome()` call never executes, so the catalog never loads. Verify with `grep -c '^async function' index.html` after edits — any count > 1 implies duplicates.