# Ultimate Scraper

Multi-source torrent scraper for Stremio. It resolves IMDB/TMDB IDs to metadata, searches torrent providers in parallel, filters bad matches, ranks streams, and optionally checks debrid cache availability.

## Quick Start

```bash
cp .env.example .env
npm install
npm start
```

Default server: `http://localhost:3091`

Run tests:

```bash
npm test
```

## Main Endpoints

| Endpoint | Description |
|----------|-------------|
| `/manifest.json` | Stremio addon manifest |
| `/stream/:type/:id.json` | Stremio stream endpoint for `movie`, `series`, or `anime` |
| `/api/stream/:type/:id.json` | API-prefixed alias for the stream endpoint |
| `/api/scrape?type=movie&id=tt0133093` | Debug/API scrape response with context and ranked results |
| `/api/rd/resolve?hash=...` | Real-Debrid direct-link resolver |
| `/api/ad/resolve?hash=...` | AllDebrid direct-link resolver |
| `/api/pm/resolve?hash=...` | Premiumize direct-link resolver |
| `/api/tb/resolve?hash=...` | TorBox direct-link resolver |
| `/api/resolve?url=...` | Proxy to the local free-scraper embed resolver |
| `/api/health` | Health check |
| `/api/health/config` | Runtime config status |

Stremio IDs supported:

```text
/stream/movie/tt0133093.json
/stream/series/tt0944947:1:1.json
/stream/movie/tmdb:603.json
/stream/series/tmdb:tv:1399:1:1.json
```

## Providers

Torrent providers are registered in `src/server.js` and normalized through `src/providers/torrent/shared.js`.

| Provider | Type | Notes |
|----------|------|-------|
| TPB | Public torrents | Uses Apibay JSON API |
| YTS | Movies | Uses YTS JSON API |
| BitSearch | Public torrents | HTML scrape |
| 1337x | Public torrents | API proxy with HTML fallback |
| NyaaSi | Anime | HTML scrape with magnet fallback |
| TokyoTosho | Anime | HTML scrape |
| AniDex | Anime | Provider module |
| SubsPlease | Anime | Provider module |
| AnimeTosho | Anime | Provider module |
| Torznab | Any | Requires Jackett/Prowlarr-compatible config |

Debrid providers are optional. Set any available API key to enable cache checks and resolve routes.

| Provider | Env Key | Cache Badge |
|----------|---------|-------------|
| Real-Debrid | `RD_API_KEY` | `RD` |
| AllDebrid | `AD_API_KEY` | `AD` |
| Premiumize | `PM_API_KEY` | `PM` |
| TorBox | `TB_API_KEY` | `TB` |

## Configuration

Copy `.env.example` to `.env` and set only the keys you need.

| Variable | Default | Purpose |
|----------|---------|---------|
| `PORT` | `3091` | HTTP server port |
| `NODE_ENV` | `development` | Runtime environment label |
| `TMDB_API_KEY` | empty | Metadata lookup for Stremio IDs |
| `JACKETT_URL` | empty | Torznab base URL, usually Jackett or Prowlarr |
| `JACKETT_API_KEY` | empty | Torznab API key |
| `RD_API_KEY` | empty | Real-Debrid token |
| `AD_API_KEY` | empty | AllDebrid token |
| `AD_API_AGENT` | `ultimate-scraper` | AllDebrid agent string |
| `PM_API_KEY` | empty | Premiumize token |
| `TB_API_KEY` | empty | TorBox token |
| `SCRAPER_TIMEOUT` | `15000` | Provider timeout in milliseconds |
| `CACHE_TTL` | `600` | SQLite scrape cache TTL in seconds |
| `FREE_SCRAPER_URL` | `http://127.0.0.1:3092` | Local free-scraper helper used by `/api/resolve` |
| `FLARESOLVERR_URL` | empty | Optional FlareSolverr URL for 1337x fallback requests |
| `SCRAPER_CONFIG_PATH` | `src/config/default-config.json` | Optional JSON config path |

JSON defaults live in `src/config/default-config.json`. Environment variables override JSON config values.

## How Results Flow

1. `parseStremioId` parses IMDB/TMDB IDs and season/episode suffixes.
2. `getMetadata` resolves title, year, aliases, and media type from TMDB.
3. `scrapeAll` queries registered torrent providers in parallel.
4. `normalizeResult` and provider helpers normalize hashes, sizes, quality, magnets, and parsed release data.
5. `matchResult` rejects samples/trailers, wrong titles, wrong years, and wrong S/E results.
6. Debrid cache checkers run concurrently for matched hashes when API keys are configured.
7. `rankResults` scores by metadata match, quality, codec, source reliability, size, seeders, debrid cache, and anime hints.
8. The stream route returns the top 30 Stremio-compatible streams.

## Project Layout

```text
ultimate-scraper/
├── .env.example
├── .gitignore
├── package.json
├── package-lock.json
├── README.md
└── src/
    ├── config.js
    ├── config/
    │   └── default-config.json
    ├── core/
    │   ├── metadataMatcher.js
    │   ├── releaseParser.js
    │   ├── resultSchema.js
    │   ├── scoring.js
    │   └── stremioId.js
    ├── providers/
    │   ├── debrid/
    │   └── torrent/
    ├── routes/
    │   ├── debrid.routes.js
    │   ├── health.routes.js
    │   └── scrape.routes.js
    ├── services/
    │   ├── cache.js
    │   ├── logger.js
    │   ├── scrapePipeline.js
    │   └── tmdb.js
    ├── server.js
    └── test/
```

Runtime SQLite cache files are created under `data/` and ignored by Git.

## Development Notes

- The server starts only when running `node src/server.js`; importing `src/server.js` in tests does not bind a port.
- `/stream/...` is the canonical Stremio route. `/api/stream/...` is kept as a debugging/API alias.
- TMDB is required for normal Stremio use because incoming IDs do not contain titles.
- Torznab is optional but recommended if you run Jackett or Prowlarr.
- The test suite uses Node's built-in test runner, no extra test framework.

## Verification

Current verified command:

```bash
npm test
```

Expected result: all tests pass.