# Addon Provider & Scraper Architecture (2026-06-01)

## Architecture: Self-Hosted Scraper Primary

Current architecture as of June 2026:

- ✅ **nexstream-scraper** (Node.js, port 3091) is the PRIMARY stream source — system addon, enabled by default, priority 15
- ✅ All old external addon APIs are proxied server-side through nexstream-scraper
- ✅ Old provider classes (CometProvider, MediaFusionProvider, TorrentioProvider) are deleted from app source
- ✅ Catalog backend REMOVED (no `CATALOG_BACKEND_URL`). Catalogs: native TMDB + Stremio addon manifests.
- ✅ Worker URLs: `ADDON_CONFIG_WORKER_URL`/`ADDON_AUTH_WORKER_URL` → `unspooled-config-v3.tw24fr.workers.dev`
- ✅ Users can also install addons manually via "Add Addon by URL" for additional sources
- ✅ AIOStreams available in registry for manual install (not auto-installed as system addon)
- ❌ No embed sources, no direct link scrapers, no hardcoded website scraper URLs
- ✅ Debrid resolution happens client-side in the app using user's own API keys

### ProviderRegistry (multi-source)

```
ProviderRegistry
├── debridRequired (only when debrid connected)
│   ├── stream-scraper (legacy — DefaultAddonScrapers.ALL is EMPTY, superseded by nexstream-scraper)
│   └── [user-installed Stremio addons via AddonRepository]
└── System addons (always loaded)
    ├── nexstream_scraper (self-hosted, enabled by default)
    ├── tmdb (native metadata catalog)
    ├── cinemeta (catalog fallback)
    └── opensubtitles
```

### Quick Check: Is the Scraper Running?

```bash
# Health check (server-side)
curl -s http://192.168.1.50:3091/health

# Manifest check
curl -s http://192.168.1.50:3091/manifest.json

# Stream check (should return streams for The Matrix)
curl -s "http://192.168.1.50:3091/stream/movie/tt0133093.json" | jq '.streams | length'

# Docker check
docker ps --filter name=nexstream-scraper
```

## Source Priority

1. **nexstream_scraper** (system addon, priority 15) — proxies Torrentio + Comet + MediaFusion + Jackett + Zilean
2. User-installed Stremio addons (via URL)
3. Debrid Search / user debrid library

## Architecture Change Log

### 2026-05-31: Self-hosted scraper introduced
- Added `services/nexstream-scraper/` — Node.js Stremio addon, debrid-agnostic, 5 parallel scrapers
- `REMOVED_SYSTEM_IDS` in `SystemAddonCatalog`: torrentio, comet, mediafusion, jackettio, knightcrawler, annatar
- `DEBRID_ENGINES` now contains only `nexstream_scraper` (enabled by default)
- All old `*Provider.kt` files deleted from app source
- Addon registry fallback still lists old addons (AIOStreams, MediaFusion, Comet, Torrentio, Jackettio) for manual install

## Addon Management

### System Addons (auto-installed)

| ID | Name | Source | Enabled |
|----|------|--------|---------|
| `nexstream_scraper` | NexStream Scraper | `BuildConfig.NEXSTREAM_SCRAPER_URL` → `192.168.1.50:3091` | yes |
| `tmdb` | TMDB Catalog | native (app-built) | yes |
| `aiometadata` | AIOMetadata (77 catalogs) | aiometadata.elfhosted.com | yes |
| `cinemeta` | Cinemeta | `v3-cinemeta.strem.io` | yes |
| `opensubtitles` | OpenSubtitles v3 | `opensubtitles-v3.strem.io` | yes |

### "Add Addon by URL" (QR-based)

The preferred way to add user addons uses QR code + phone browser:

1. User clicks "📱 Scan QR" on the addon screen
2. TV shows QR code (via QrCodeGenerator + DeviceIpAddress)
3. Phone scans QR → opens Cloudflare Worker form
4. User pastes manifest URL → Worker validates → TV polls and installs

**No keyboard needed on TV** for any addon operations.
3. TV shows QR code + user code
4. Phone scans QR → visits `real-debrid.com/device` → enters code → authorizes
5. Worker polls `POST /token` until granted
6. TV polls `GET /debrid/auth/status/{code}` → gets token → connects via `DebridService.connect()`

### API Key Entry — TorBox, AllDebrid, Debrider, Offcloud

1. TV calls Cloudflare Worker `GET /debrid/key/start/{service}`
2. Worker creates DO session, TV shows QR
3. Phone scans QR → opens API key entry form
4. User pastes key → POST to `/debrid/auth/submit/{code}`
5. TV polls → gets key → connects

## Cloudflare Worker: addon-config-worker

Location: `~/Unspooled/services/addon-config-worker/`

**DO classes:**
- `ConfigSessionDO` — Addon configuration sessions (QR → phone configures → TV installs)
- `AuthSessionDO` — Debrid auth sessions (OAuth device code + API key entry)
- `AddonUrlSessionDO` — Addon URL entry sessions

**Key routes:**
- `/addons/config/{code}` — Addon configuration (existing)
- `/debrid/oauth/start/{service}` — OAuth device code start
- `/debrid/key/start/{service}` — API key session start
- `/debrid/auth/form/{code}` — Auth form HTML page
- `/debrid/auth/status/{code}` — TV polls for auth result
- `/addon/url/start` — Create addon URL entry session
- `/addon/url/form/{code}` — Addon URL entry HTML page
- `/addon/url/status/{code}` — TV polls for URL result

**Deploy:** `cd services/addon-config-worker && npx wrangler deploy`

Requires secrets set via `npx wrangler secret put BACKEND_SECRET` and `PUBLIC_HOST` in `wrangler.toml`.

## Source Priority (clean)

1. AIOStreams configuredManifestUrl (auto-configured with debrid)
2. User-installed Stremio addons (via URL)
3. Debrid Search / user debrid library
4. *(No embed/webpage/scraper fallback)*

## Verification

### Provider Registry Test

`app/src/test/java/.../ProviderRegistryProvidedSourcesTest.kt` verifies:

- No embed/direct-link scraper providers registered
- No providers available without debrid connection
- No banned scraper URLs in any provider config
- No always-enabled free tier

Run: `./gradlew :app:testDebugUnitTest --tests "*ProviderRegistryProvidedSourcesTest"`
- `id: com.aiostreams.viren070`, `version: 2.30.2`
- `resources: []` (empty — addon only serves resources after user configuration)
- `configurationRequired: true`
- Stream endpoint returns error stream: "AIOStreams v2 requires you to reconfigure" until configured
- `priority = 15` (highest in app)

### MediaFusion (`MediaFusionProvider.kt`)
- `BASE_URL = "https://mediafusion.elfhosted.com"` — correct, addon at root
- `id: stremio.addons.mediafusion|elfhosted`, `version: 5.5.2`
- `resources: [catalog, {name: "stream", types: [movie, series, tv, events], idPrefixes: [tt, tmdb:, tvdb:, mal:, mf, dl]}]`
- Manifest: ✅ 200, 683B, 0.27s
- Stream endpoint: ✅ 200 but empty `{"streams":[]}` until debrid configured
- `priority = 7`

## Removed Providers

### ~~Comet~~ (`CometProvider.kt`) — Removed 2026-05-28
- `BASE_URL = "https://comet.elfhosted.com"` — manifest returned 200, but stream endpoint returned **403 Forbidden** (Cloudflare WAF)
- Blocked all non-browser requests even with full UA/Accept/Origin/Referer headers — curl and app's OkHttp both likely blocked
- `id: comet.elfhosted.com`, `version: 2.0.0`
- `resources: [{name: "stream", types: [movie, series], idPrefixes: [tt, kitsu]}]`
- Project active (g0ldyy/comet, 967 stars) but elfhosted deployment blocks all unauthenticated stream requests
- Redundant with AIOStreams (same 17+ scrapers, same debrid coverage)
- Removed from: `CometProvider.kt` (deleted), `ProviderRegistry.kt`, `AddonMarketplace.kt`, `AddonRegistryRepository.kt`, `AddonsViewModel.kt`
- Presets updated: "balanced" → AIOStreams + MediaFusion, "max" → AIOStreams + MediaFusion + Torrentio + Jackettio

## Protocol Details

### Stremio Addon Manifest Fields to Check
- `id`: unique addon identifier
- `resources`: array of `{name: "stream", types: [...], idPrefixes: [...]}`
- `types`: supported content types (movie, series, etc.)
- `behaviorHints.configurable` / `configurationRequired`
- `catalogs`: array if addon provides browse catalogs

### Stream Response Fields
```json
{
  "streams": [
    {
      "url": "https://...",           // direct URL
      "externalUrl": "https://...",   // opens in browser
      "infoHash": "...",              // torrent info hash
      "fileIdx": 0,                   // file index in torrent
      "behaviorHints": { ... }        // player hints
    }
  ]
}
```

### Common Failure Modes
| Symptom | Likely Cause |
|---------|-------------|
| Manifest returns PWA/HTML | Wrong BASE_URL (addon at subpath like `/stremio/`) |
| Manifest 200 but no `id`/`resources` | URL serves non-Stremio content (PWA, HTML page) |
| Stream 403 (Cloudflare) | Elfhosted WAF blocking unconfigured requests |
| Stream 200, empty `streams[]` | Addon needs debrid service config |
| Stream 200, error streams | Addon version mismatch or needs reconfig |
| Slow/Timeout | Addon is heavy (MediaFusion uses Redis/PG) |

## Addons Screen Architecture

Two implementations, only ONE wired in NavHost at a time:

| Screen | File | LOC | Status | Style |
|--------|------|-----|--------|-------|
| `AddonControlScreen` | `ui/screens/addons/AddonControlScreen.kt` | 326 | WAS wired (old, pre-2026-05-28) | Sectioned list, presets, health buttons, dark bg |
| `AddonsScreen` | `ui/screens/addons/AddonsScreen.kt` | 1107 | NOW wired (new, since 2026-05-28) | 5-tab, NuvioTheme, QR mode, manifest preview |

Switch by editing `NexStreamNavHost.kt` at `Screen.Addons` route:
```kotlin
// composable(route = Screen.Addons.route) {
//   Old: AddonControlScreen()
//   New: AddonsScreen(navController = navController)  // ← currently active
// }
```

### AddonsScreen Features
- 3 addon categories: system → debrid → user (sectioned)
- 5 tabs: Installed, Discover, Priority, Updates, Advanced
- QR code phone pairing (`AddonConfigServer` — NanoHTTPD)
- Safe mode toggle (prevents disabling system addons)
- URL input with manifest preview before install
- Shimmer loading skeletons
- Error banners (dismissible)
- Health snapshots (response time, success rate, timeout count, auto-disable)
- Update checking tab

### Upgrade Issue: Safe Mode + System Addons
When safe mode is on, system addons cannot be toggled off. The code at `AddonsScreen.kt` line 337-340 explicitly blocks this. If a user needs to disable a system addon, safe mode must be turned off first.