# Go Scraper Server — Architecture & Cloudflare Bypass

## When to use this approach
Building a self-hosted Go scraper server (non-Stremio) that scrapes movie/TV streaming sites, resolves embed URLs to m3u8, and serves a web frontend. A full-stack Go alternative to the Node.js Stremio addon pipeline.

## Architecture

```
Browser (user) → Go Server (Fiber) → TMDB API (catalog metadata)
                                   → Site Providers (scrape movie sites for embed URLs)
                                   → Extractors (resolve embed → m3u8)
                                   → nodriver (Python headless Chrome, Cloudflare bypass)
                                   → HLS proxy (proxied manifest/segment serving)
```

## CRITICAL: Site-Specific ID Flow

**This is the #1 mistake when porting Android scrapers to Go.** The original Android app flows IDs like this:

```
Provider's own search → returns site-specific ID (e.g., "/watch-movie-27205")
     ↓
getMovie(siteID) → scrapes movie page
     ↓
getServers(siteID, videoType) → uses site's numerical ID for AJAX endpoint
     ↓
getVideo(server) → resolves embed URL to m3u8
```

**You CANNOT pass TMDB IDs to site providers.** TMDB ID `27205` (Inception) is meaningless to Sflix, CineCalidad, etc. Each provider needs its own slug format. The correct flow:

```
1. TMDB catalog (for browsing/search) — uses TMDB IDs
2. User clicks movie → search EACH provider for the movie TITLE
3. Extract provider's own ID/slug from search results
4. Call GetServers(providerID) with that site-specific ID
5. Resolve embed URLs to m3u8
```

### One-click play auto-flow
When building a streaming frontend, clicking a movie should auto-find sources and play the first one:
```
1. User clicks movie in TMDB catalog
2. Frontend calls `/links/movie/{id}?title={encoded_title}`
3. Server searches each provider for the title, gets site-specific IDs, scrapes embed URLs
4. Frontend auto-plays the first embed source as an iframe (user's browser handles Cloudflare)
5. Source list shown below for switching
```

## Cloudflare Bypass Strategy

**The key problem**: Go's net/http + uTLS cannot pass Cloudflare JS challenges. Android apps (OkHttp/Conscrypt) work because they run on real devices with real TLS fingerprints. Server-side bypass REQUIRES a real browser engine.

**2026 state of Cloudflare bypass tools**: nodriver > chromedp > FlareSolverr
- **nodriver** (github.com/ultrafunkamsterdam/nodriver): Successor to undetected-chromedriver. **28/31 Cloudflare targets passed** in benchmark, ZERO blocked. Direct CDP WebSocket — no Playwright/Selenium middleware. Python library.
- **chromedp**: Go library for Chrome DevTools Protocol. Requires NoSandbox config. Slower than nodriver.
- **FlareSolverr**: Python/Docker (14.2k ⭐). Last release may be stale. Uses Selenium + undetected-chromedriver.
- **The real fix for all**: headless browser. No TLS trick or HTTP library bypasses Cloudflare JS challenges in 2026.

### Dual-path HTTP client pattern

```go
func fetchPage(url string) ([]byte, error) {
    body, err := tryDirectFetch(url)
    if err == nil { return body, nil }
    if isCloudflareError(err) {
        return nodriverFetch(url, 10*time.Second)
    }
    return nil, err
}
```

### nodriver subprocess integration (Go → Python)

Create a Python venv with nodriver, call as subprocess:
```bash
python3 -m venv .venv-nodriver
.venv-nodriver/bin/pip install nodriver
```

```go
func nodriverFetch(url string, timeout time.Duration) ([]byte, error) {
    cmd := exec.Command(
        "/path/to/.venv-nodriver/bin/python",
        "/path/to/nodriver_fetch.py",
        url, fmt.Sprintf("%.0f", timeout.Seconds()),
    )
    output, err := cmd.Output()
    // Parse JSON result with {success, html, html_length, error}
}

func isCloudflareError(err error) bool {
    errStr := err.Error()
    return strings.Contains(errStr, "malformed HTTP response") ||
        strings.Contains(errStr, "HTTP/1.x transport connection broken") ||
        strings.Contains(errStr, "403") || strings.Contains(errStr, "cloudflare") ||
        strings.Contains(errStr, "handshake")
}
```

### nodriver Python script pattern (`nodriver_fetch.py`)

```python
import nodriver as nd, asyncio, sys, json

async def fetch(url: str) -> str:
    browser = await nd.start(headless=True, sandbox=False, disable_gpu=True)
    try:
        tab = await browser.get(url)
        await tab.wait_for("body", timeout=15)
        await asyncio.sleep(3)
        html = await tab.evaluate("document.documentElement.outerHTML")
        return html
    finally:
        await browser.aclose()

if __name__ == "__main__":
    html = asyncio.run(fetch(sys.argv[1]))
    print(json.dumps({"success": True, "html_length": len(html), "html": html}))
```

Key nodriver API notes:
- `browser.get(url)` returns a `Tab` — NOT `page` or `tab.content()`
- Get HTML: `await tab.evaluate("document.documentElement.outerHTML")` NOT `tab.content()`
- Close: `await browser.aclose()` NOT `browser.close()`
- `evaluate()` returns CDP-serialized values; complex objects need `JSON.stringify()`
- Always use `sandbox=False` flag when running as non-root

### uTLS Configuration

```go
uconn := utls.UClient(tcpConn, &utls.Config{
    ServerName: host,
    NextProtos: []string{"http/1.1"},   // NO h2 — Go can't handle HTTP/2 over uTLS
}, utls.HelloChrome_120)

transport := &http.Transport{
    DialTLS:          dialTLS,
    ForceAttemptHTTP2: false,
    TLSNextProto:     make(map[string]func(...) http.RoundTripper),
}
```

Without disabling HTTP/2, uTLS negotiates h2 via ALPN but Go's transport can't parse it, causing "malformed HTTP response" errors with binary content like `\x00\x00\x12\x04...` (Cloudflare HTTP/2 connection preface).

## Porting Android (Kotlin) scrapers to Go

### Original Streamflix Kotlin provider structure

The original APK uses Retrofit with JsoupConverterFactory — a custom converter mapping CSS selectors to function return types:

```kotlin
private interface SflixService {
    @GET("home")
    suspend fun getHome(): Document

    @GET("ajax/episode/list/{id}")
    suspend fun getMovieServers(@Path("id") id: String): Document

    @GET("ajax/episode/sources/{id}")
    suspend fun getLink(@Path("id") id: String): Link

    @GET
    suspend fun getEmbed(@Url url: String): Embed
}

val client = OkHttpClient.Builder()
    .readTimeout(30, TimeUnit.SECONDS)
    .dns(DnsResolver.doh)
    .build()
```

When porting to Go:
1. Replace Jsoup CSS selectors with `regexp` or `goquery` equivalents
2. Parse the original `.kt` for `.select("selector")`, `.selectFirst("selector")`, `.attr("attr")` calls
3. Handle both HTML responses (Jsoup Document) and JSON responses (Gson) separately

### Complete playback flow (from Streamflix PlayerViewModel)

```
1. PlayerViewModel(videoType, id) created with site-specific ID
2. init() calls getServers(videoType, id) — provider.getServers(id, videoType)
3. For Movie: GET /ajax/episode/list/{numericalId} → HTML with <a> elements
4. Each <a> has data-id attribute → Video.Server(id, name)
5. User clicks play → getVideo(server):
   a. GET /ajax/episode/sources/{serverId} → JSON {link, type, sources, tracks}
   b. Extractor.extract(link.link, server) → resolves embed URL → m3u8
6. Video plays in ExoPlayer (Android) or hls.js (web)
```

### Common pitfalls when porting

1. **Site-specific IDs** — NEVER pass TMDB IDs to site providers.
2. **TLS fingerprint mismatch**: Android OkHttp uses Conscrypt. Go's uTLS doesn't match Chrome closely enough.
3. **CSS selector → regex**: Parse the .kt file for `.select()` and `.selectFirst()` calls.
4. **Retrofit/JsoupConverterFactory**: Kotlin Retrofit + custom HTML converter. Go needs regex or goquery.
5. **Coroutine → goroutine**: Kotlin `suspend fun` → Go goroutines with `sync.WaitGroup` + context.
6. **DNS-over-HTTPS**: Original may use OkHttp's DnsOverHttps.
7. **Android-specific APIs**: `android.util.Base64`, `android.util.Log`, WebView need JVM/stdlib replacements.
8. **Language-specific titles**: Spanish sites don't have English titles. "Inception" → "Origen" in Spanish.
9. **Site URLs change**: CineCalidad moved from `.ec` to `.am`. Verify current domains.
10. **Dead sites**: sflix.to returns 522 — completely dead. Test each provider individually.

### Parallel porting strategy

Use `delegate_task` subagents for 50+ extractors in parallel. Each subagent needs Kotlin source files, existing Go framework (interface, HTTP client, crypto), target directory, and clear pattern examples. Complex extractors with custom crypto need separate handling.

### Testing each provider

Test individually with `/api/test/provider/{Name}/{id}`. Check server logs for nodriver output. Use movie titles the site actually carries.

## Streaming Site Frontend

### Playback flow

1. User browses TMDB catalog
2. Clicks movie → detail view → auto-finds sources → auto-plays first source
3. Source loads as iframe embed (user's browser handles Cloudflare)
4. Background: try server-side resolve to m3u8; show "▶ Direct" button if found
5. Manual: paste any m3u8 URL → hls.js player handles it

### Key design decisions

- Iframe embeds are PRIMARY — server-side Cloudflare bypass is secondary bonus
- TMDB for catalog — site scrapers for source discovery only
- hls.js for direct m3u8 with custom header support via xhrSetup
- Manual URL paste as fallback
- Auto-play first source on detail page open — true one-click experience
- Site providers are unreliable — always have fallbacks

### Real-Debrid Integration (Browser Testing)

For quick RD testing (checking cache + playing) without building a full app, use a Python proxy server pattern:

### RD API Flow

```
1. Check cache: GET /torrents/instantAvailability/{infoHash}
   Response: { "HASH": [{"rd": [...variants]}] } or {} if uncached
   
2. Add magnet: POST /torrents/addMagnet  
   Body: magnet=urn:btih:{HASH}
   Response: { "id": "torrent_id" }
   
3. Select files: POST /torrents/selectFiles/{id}
   Body: files=all
   
4. Wait for download: GET /torrents/info/{id}
   Status becomes "downloaded" when ready
   
5. Get playable link: POST /unrestrict/link
   Body: link={original_link}
   Response: { "download": "https://...direct.m3u8" }
```

### CORS Proxy Pattern

RD API doesn't send CORS headers. Browser JS cannot call RD directly. Use a local Python proxy:

```python
class ProxyHandler(http.server.SimpleHTTPRequestHandler):
    def do_GET(self):
        if self.path.startswith('/rd/'):      # proxy to RD API
            url = 'https://api.real-debrid.com/rest/1.0/' + self.path[4:]
            req = urllib.request.Request(url)
            req.add_header('Authorization', f'Bearer {RD_KEY}')
            # proxy response back with CORS headers
        if self.path.startswith('/rd-post/'):  # POST to RD API
            # same but POST method
        if self.path.startswith('/scraper/'):  # proxy to nexstream-scraper or Go scraper
            url = 'http://localhost:3091/' + self.path[9:]
        # serve static files otherwise
```

### Quick test flow (browser)

```
1. HTML page fetches streams from scraper via /scraper/
2. Checks RD cache via /rd/torrents/instantAvailability/{hash}
3. Shows cached/uncached badges
4. User clicks ▶ on cached source
5. POST /rd-post/torrents/addMagnet → /rd-post/selectFiles → poll /rd/torrents/info/
6. POST /rd-post/unrestrict/link → get playable m3u8
7. Feed to hls.js → video plays
```

### RD API Key Safety

Never hardcode RD keys in source files. For quick testing:
- Pass via server-side env var (Python proxy keeps the key server-side)
- The key NEVER reaches the browser — the proxy adds Authorization header
- Do NOT persist to config files, .env, or commit

### nexstream-scraper (Stremio-Compatible Alternative)

The nexstream-scraper at `~/Unspooled/services/nexstream-scraper/` is a Node.js Stremio addon that scrapes Torrentio, Comet, MediaFusion for torrent sources. Returns info hashes + magnet links + quality + seeders. Debrid resolution is client-side (correct architecture).

Routes:
- `GET /health` — dashboard with cache + per-scraper health
- `GET /manifest.json` — Stremio manifest
- `GET /stream/movie/{imdbId}.json` — Stremio format streams
- `GET /stream/raw/movie/{imdbId}.json` — raw pipeline output
- `GET /stream/:type/:id.json?debug=true` — includes _debug section

Key difference from Go scraper: nexstream-scraper returns torrent hashes (not m3u8). Debrid resolution happens in the app. This is the correct architecture — server should never hold user debrid tokens.

## Duplicate function bug warning
When doing multiple frontend patches, `async function foo() {}` definitions can duplicate. JS silently stops after redeclaration error, making page appear broken. After edits, `grep -n '^async function' index.html` to verify no duplicates.