# Parallel Porting Pattern — Kotlin Extractors to Go

When porting many (80+) similar Kotlin extractors to Go, this pattern worked:

## Batch Strategy

Split the work into 3-4 batches sent via `delegate_task`:

| Batch | What | Subagent context | Outcome |
|---|---|---|---|
| **Complex crypto** | Rabbitstream, Megacloud, Vidsrc.to, Vidplay, Filemoon, VixSrc, Chillx | Full Kotlin file(s) + existing Go interfaces + crypto package API | 4-7 Go files, each with full AES/RC4/MD5/ECDSA decryption |
| **Simple regex** | 60+ extractors (Vidoza, StreamWish, MixDrop, etc.) | Interface definitions + crypto API + list of domains to port + generic template code | Single `simple.go` file with all 60+ registered |
| **Site providers** | CineCalidad, Cuevana, Pelisplusto, TMDB, AnimeFLV, etc. | Provider interface + existing Sflix example | 4-9 Go provider files |

## Key to Success

1. **Provide the exact interface** in each subagent's context (Extractor or Provider interface)
2. **Provide the crypto API** — list available functions with signatures
3. **For simple extractors: give the template** — subagents fill in domain name + URL extraction logic
4. **3 concurrent max** — limited by `delegation.max_concurrent_children`
5. **Build after each batch** — verify compile before launching next batch

## Generic Template for Simple Extractors

```go
type XxxExtractor struct{}
func (e *XxxExtractor) Name() string    { return "Xxx" }
func (e *XxxExtractor) MainURL() string { return "xxx.com" }
func (e *XxxExtractor) Extract(link string) (*models.Video, error) {
    body, err := fetchString(link)
    if err != nil { return nil, err }
    if m := m3u8Regex.FindString(body); m != "" { return MakeVideo(m, nil), nil }
    // Try packed JS
    result := UnpackJs(body)
    if result.Success {
        if m := m3u8Regex.FindString(result.Data); m != "" { return MakeVideo(m, nil), nil }
    }
    return nil, fmt.Errorf("no source")
}
func init() { Register(&XxxExtractor{}) }
```

## Reality Check: What Actually Works After Porting

Once the port is done and 80+ extractors are registered, **most embed sources will still fail to produce m3u8 URLs** because:

1. **Embed aggregator URLs are dead pages** — The 78 embed source templates (VidSrc.icu, AutoEmbed, 2Embed, etc.) point to aggregator sites that are either abandoned, behind Cloudflare, or return redirects to dead pages. The `/api/resolve` endpoint returns "no extractor found" because these aren't video hoster domains — they're aggregator fronts that need their own scraping.

2. **Site scrapers are blocked by Cloudflare** — Sflix, CineCalidad, etc. all hit Cloudflare challenge pages. The uTLS client helps but doesn't fully bypass Cloudflare's JavaScript challenges. The HTTP client gets garbled binary responses (HTTP/2 frames parsed as HTTP/1.x).

3. **Extractors only work for known hoster domains** — voe.sx, streamtape.com, mixdrop.co, etc. The extractors can resolve those to m3u8, but only IF those hoster URLs appear in the first place (they don't from dead aggregators).

### uTLS + Cloudflare: The HTTP/2 Transport Mismatch

The `\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00d` malformed response is **Cloudflare's HTTP/2 connection preface** being received by an HTTP/1.x transport. Here's why this happens:

1. uTLS negotiates a TLS connection advertising `h2` in ALPN (NextProtos)
2. Cloudflare sees h2 in the handshake and switches to HTTP/2 mode
3. The HTTP/2 connection preface (binary bytes starting with `0x00...`) is sent to the client
4. Go's `http.Transport` expects HTTP/1.1 text and fails with "malformed HTTP response"

**The fix — keep transport strictly HTTP/1.1:**

```go
// In the uTLS dialer:
uconn := utls.UClient(tcpConn, &utls.Config{
    InsecureSkipVerify: false,
    ServerName:         host,
    NextProtos:         []string{"http/1.1"},  // NO "h2"
}, utls.HelloChrome_120)

// In the transport:
transport := &http.Transport{
    DialTLS:            dialTLS,
    ForceAttemptHTTP2:  false,
    TLSNextProto: make(map[string]func(authority string, c *tls.Conn) http.RoundTripper),
}
```

**Why this works:** Even though real Chrome ALPN includes `h2`, Cloudflare won't reject a Chrome TLS fingerprint that advertises `http/1.1` only. The TLS fingerprint (JA3/JA4) is what matters for bypass, and `HelloChrome_120` still produces a valid Chrome fingerprint even with a modified ALPN list.

**When this fix is insufficient:** Sites using Cloudflare Turnstile/Challenge (JavaScript) will still block because they need JS execution. The only reliable server-side bypass is a headless browser (chromedp/playwright). For the frontend, **iframe embeds** are the practical fallback — the user's browser handles the JS challenge natively.

### Streaming Site Architecture — Reference Pattern

When server-side scraping is Cloudflare-blocked, this two-tier playback architecture proved reliable:

```
TMDB API ─→ Catalog browsing (search, trending, popular, details) ✅
                   │
                   ▼
           Find Sources (get embed URLs)
                   │
          ┌────────┴────────┐
          ▼                  ▼
    /api/resolve        iframe embed
    (server-side)       (browser handles CF)
          │                  │
    m3u8 found? ──yes──▶ playVideo(hls.js)
          │
          no
          ▼
    playEmbed(iframe)
```

Key principles:
- **TMDB API is the catalog** — works 100% reliably, no scraping needed for discovery
- **Iframe embeds are the primary playback path** — resolve is a bonus optimization
- **Resolve runs in background** — start the iframe immediately, try resolve after
- **Manual URL paste** — always include a text input for users to paste their own m3u8/embed URLs
- **HLS proxy ready** — `/proxy/hls/manifest?url=` and `/proxy/hls/segment?url=` endpoints for streaming through the server with auth headers

### Sflix Debugging: Endpoint Discovery

The Kotlin `SflixProvider` uses these exact Retrofit endpoints:

| Endpoint | Returns | Purpose |
|---|---|---|
| `GET /home` | HTML Document | Homepage with featured, trending, latest sections |
| `GET /ajax/episode/list/{id}` | HTML Document | Episode list for movie (returns server list in HTML) |
| `GET /ajax/episode/sources/{id}` | JSON `{type, link, sources, tracks}` | **This is the critical endpoint** — returns the actual embed URL in the `link` field |
| `GET` (the embed URL from sources) | JSON `{sources: [{file, type}], tracks: [{file, label, kind}]}` | Returns the final m3u8 URLs |

The Kotlin code hits `ajax/episode/sources/{id}` (NOT `ajax/episode/list/{id}`) to get the embed link. If your Go port only hits the list endpoint, you're getting HTML instead of the JSON embed response.

### Real Extractors vs Registration Count

**83 registered extractors ≠ 83 working extractors.** Registration is the easy part. Most registered domains won't actually resolve because:
- The embed pages they'd scrape are dead/blocked
- Some require WebView/JS execution (StreamWish, Upzone)
- Some have complex ECDSA attestation flows (Filemoon) that were simplified in the Go port
- The original Kotlin extractors ran on Android with real device TLS + fingerprinting

**Measure success by resolve rate, not registration count.** For a realistic test, use known-working hoster URLs (not embed aggregator URLs). The most reliable test is to paste a known m3u8 URL into the player directly.

## Pitfalls

- **Subagent API key auth failures** — subagents inherit the parent's model/provider. If auth fails, the subagent dies immediately. Pin a stable model for subagents via `delegation.model` in config. This session a subagent failed immediately with "Authentication Fails, Your api key ****red. is invalid" — verify subagent auth before dispatching large batches.
- **Large contexts** — each subagent gets a full copy of existing Go code as context. The simple extractors batch had ~1.8M input tokens. This is normal and works but takes ~4-5 minutes per batch.
- **Max concurrent children = 3** for this Hermes config. Larger batches must be written in bigger subagent tasks (one agent handles more per run).
- **Build errors are expected** on the first attempt. Each subagent's Go code may have minor issues (missing imports, wrong type names). These are faster to fix manually than to re-dispatch. In this session the simple.go file needed minor patches after generation — common and expected.
- **Fiber `c.Params("*")` strips query strings** — When proxying to an API like TMDB, the wildcard route param (`/api/tmdb/*`) does NOT include query parameters. Always rebuild query strings from `c.Request().URI().QueryString()` when building upstream URLs. Without this, `search/multi?query=inception` becomes just `search/multi` and returns 0 results.
- **Return JSON, not HTTP errors** — When a resolve fails (dead embed, blocked site), return `{"success": false, "error": "..."}` with HTTP 200, not a 502/400 status. The frontend JS handles error states natively; HTTP error codes trigger browser console noise and CORS issues. The frontend should fall back to iframe embed on any resolve failure.
- **83 registered extractors ≠ 83 working extractors** — Registration is the easy part. Most registered domains won't actually resolve because the embed pages they'd scrape are dead. Measure success by resolve rate, not registration count.