# Ultimate Scraper v5 — Streamflix→Go Port Reference

## Overview

Full Go port of the Streamflix Android scraper engine (Kotlin) merged into the Ultimate Scraper Go server. 83 extractors + 9 site providers + crypto layer + HTTP server + streaming frontend.

## Server: `/home/rurouni/ultimate-scraper/` (Debian)

### Routes
| Endpoint | Description |
|---|---|
| `GET /` | Netflix-style streaming site (TMDB catalog + one-click play) |
| `GET /test` | Developer test dashboard (extractors, providers, resolve) |
| `GET /health` | Health check + cache stats |
| `GET /providers` | List all registered site providers |
| `GET /links/movie/:id` | Resolve movie by ID through ALL providers (parallel goroutines) |
| `GET /links/series/:id/:season/:episode` | Resolve TV episode |
| `GET /api/resolve?url=&browser=true` | Resolve single embed URL (with optional chromedp fallback) |
| `GET /api/test/provider/:name/:id` | Test a specific provider |
| `GET /api/tmdb/*` | TMDB API proxy (search, details, trending, etc.) |
| `GET /proxy/hls/manifest?url=` | HLS manifest proxy |
| `GET /proxy/hls/segment?url=` | HLS segment proxy |

### Run
```bash
cd /home/rurouni/ultimate-scraper && ./ultimate-scraper   # binary on :8099
# or: docker compose up -d
```

## Extractors — 83 Registered

### Complex (individual files with full crypto)
- `rabbitstream.go` — Rabbitstream, Megacloud, Dokicloud, PremiumEmbeding (AES-CBC with MD5 KDF, player JS key extraction)
- `embed.go` — RabbitStream, VidPlay, MyCloud, VidsrcTo (RC4), VidsrcNet (12 decrypt variants), TwoEmbed, VixSrc
- `common.go` — Voe (7-step decrypt chain), StreamTape, MixDrop, DoodStream, FileMoon, Uqload, StreamSB, GoVid, SuperVideo, VixCloud, StreamWish, VidGuard, StreamHub, VTube, Vidoza

### Simple (all 60+ in `simple.go`)
BigWarp, LoadX, VidHide, Ridoo, USTR, Okru, Goodstream, Lamovie, MailRu, Dropload, Rpmvid, YourUpload, VidPly, StreamUp, GxPlayer, Maxstream, PlusPomla, Oneupload, Fsvid, GoogleDrive, Pcloud, AmazonDrive, Vidzy, Gupload, VidLink, Vidflix, Vidrock, Videasy, Vidzee, Vidnest, PrimeSrc, Vidora, Dailymotion, ApiVoirFilm, Streamix, ShareCloudy, Streamruby, Vidara, Vidsonic, Hxfile, Zilla, PDrain, VidxGo, LuluVdo, MagaSavor, VidMoLy, VideoSibNet, SaveFiles, Einschalten, Frembed, Moflix, MStreamDay, StreamhubTo, Vtbe, Upzone, Closeload, VidsrcRu, VidozaAlias, Veev, VidsrcNetAdvanced

## Site Providers — 9 Registered

| Provider | Language | Type |
|---|---|---|
| TMDB (TmdbProvider) | en | Metadata/catalog API |
| Sflix (SflixProvider) | en | Movie/TV site scraper |
| RidoMovies (RidomoviesProvider) | en | Movie/TV site scraper |
| CineCalidad (CineCalidadProvider) | es | Spanish movie site |
| Cuevana 3 (CuevaProvider) | es | Spanish movie/series site (FastAPI + HTML fallback) |
| Pelisplusto (PelisplustoProvider) | es | Spanish movies/TV |
| AnimeFLV (AnimeFlvProvider) | es | Anime site |
| AnimeSaturn (AnimeSaturnProvider) | it | Italian anime site |
| AnimeUnity (AnimeUnityProvider) | it | Italian anime site (JSON-in-HTML parsing) |

## Key Crypto Ports
- AES-CBC/PKCS5 with MD5 KDF (Rabbitstream, Chillx) — OpenSSL EVP_BytesToKey style
- AES-GCM (Filemoon ECDSA attestation flow)
- RC4 KSA/PRGA (Vidsrc.to, Vidplay)
- JsUnpacker P.A.C.K.E.R. deobfuscation
- Voe 7-step decrypt chain: ROT13 → pattern replace → base64 → char shift -3 → reverse → base64
- Vidsrc.net 12-method decrypt variants (chunk reverse, hex XOR, ROT13, Caesar, URL-safe base64)
- AesHelper MD5 KDF with salt+iv from JSON

## Frontend: `/home/rurouni/ultimate-scraper/frontend/`

| File | Purpose |
|---|---|
| `index.html` | Netflix-style streaming site with TMDB catalog, detail view, one-click embed play, manual URL input, hls.js player |
| `test.html` | Developer test dashboard with provider tests, extractor tests, server health, TMDB search |

## Additional Scraper Repos (on CachyOS at `/home/raymond/Desktop/Scaraper repors/`)

| Repo | Purpose |
|---|---|
| `FlareSolverr-3.5.0/` | Python/Docker Cloudflare bypass proxy (industry standard) |
| `vidsrc-bypass-main/` | Bypasses vidsrc embed sources |
| `mediaflow-proxy-light-1.1.1/` | Media proxy |
| `agentic-stealth-browser-2.4.0/` | Stealth browser for scraping |
| `foxhound-0.0.26/` | Browser-based scraping tool |
| `yt-dlp-2026.03.17/` | YouTube-dl fork (extracts m3u8 from many sources) |
| `iframe-extractor-main/` | Extracts video from iframe embeds |
| `Scrapling-main/` | Python scraping framework |
| `CyberScraper-2077-main/` | AI-powered scraper |

## ID Flow Criticality

**The single most important lesson**: Site scrapers need SITE-SPECIFIC IDs, not TMDB IDs. The original APK flows IDs from the provider's own search/catalog. When porting, you MUST search the provider first to get the correct site ID:

```go
// WRONG — passes TMDB ID directly
provider.GetServers("27205")  // TMDB ID — site doesn't recognize it

// RIGHT — search provider for title first
results, _ := provider.Search("Inception", 1)
siteID := results[0].ID  // e.g., "/watch-movie-27205"
provider.GetServers(siteID)  // works!
```

## Project Stats
- 41 Go files, 16,517 lines of Go
- 12MB stripped binary
- `go build ./...` and `go vet ./...` clean
- Docker with distroless, 128MB RAM limit, healthcheck
- Headless Chromium v149 installed for Cloudflare bypass
- chromedp integrated for browser-based fallback when direct HTTP is blocked