[Skip to content](https://github.com/peterdsp/Magnetio/tree/main/scraper#start-of-content) You signed in with another tab or window. [Reload](https://github.com/peterdsp/Magnetio/tree/main/scraper) to refresh your session.You signed out in another tab or window. [Reload](https://github.com/peterdsp/Magnetio/tree/main/scraper) to refresh your session.You switched accounts on another tab or window. [Reload](https://github.com/peterdsp/Magnetio/tree/main/scraper) to refresh your session.Dismiss alert {{ message }} [peterdsp](https://github.com/peterdsp)/ **[Magnetio](https://github.com/peterdsp/Magnetio)** Public - [Notifications](https://github.com/login?return_to=%2Fpeterdsp%2FMagnetio) You must be signed in to change notification settings - [Fork\\ 3](https://github.com/login?return_to=%2Fpeterdsp%2FMagnetio) - [Star\\ 19](https://github.com/login?return_to=%2Fpeterdsp%2FMagnetio) ## Collapse file tree ## Files main Search this repository(forward slash)` forward slash/` / # scraper / Copy path ## Directory actions ## More options More options ## Directory actions ## More options More options ## Latest commit [![dependabot[bot]](https://avatars.githubusercontent.com/in/29110?v=4&size=40)](https://github.com/apps/dependabot)[dependabot\[bot\]](https://github.com/peterdsp/Magnetio/commits?author=dependabot%5Bbot%5D) [Bump the scraper-minor-patch group across 1 directory with 2 updates (](https://github.com/peterdsp/Magnetio/commit/c5dfb24bb5a51e9ffb99bda9eeba0f25825176e7) [#…](https://github.com/peterdsp/Magnetio/pull/95) Open commit detailssuccess last weekJun 25, 2026 [c5dfb24](https://github.com/peterdsp/Magnetio/commit/c5dfb24bb5a51e9ffb99bda9eeba0f25825176e7) · last weekJun 25, 2026 ## History [History](https://github.com/peterdsp/Magnetio/commits/main/scraper) Open commit details [View commit history for this file.](https://github.com/peterdsp/Magnetio/commits/main/scraper) History / # scraper / Copy path Top ## Folders and files | Name | Name | Last commit message | Last commit date | | --- | --- | --- | --- | | ### parent directory
[..](https://github.com/peterdsp/Magnetio/tree/main) | | [lib](https://github.com/peterdsp/Magnetio/tree/main/scraper/lib "lib") | [lib](https://github.com/peterdsp/Magnetio/tree/main/scraper/lib "lib") | [Fix scraper 429 rotation, add Debrid catalog toggle, document HTTPS/O…](https://github.com/peterdsp/Magnetio/commit/797dfb4886d061906069c51e344f4ec5ecaea99e "Fix scraper 429 rotation, add Debrid catalog toggle, document HTTPS/Oracle setup (#91) * Scraper resilience, Debrid catalog toggle, HTTPS/Oracle deploy docs Scraper (#65): - Rotate domains on HTTP 429 (previously bubbled up as an unhandled status, never tried sibling domains) - Longer cooldown when a domain is rate-limited vs hard-failed - Bumped per-provider timeout from 15s to 25s; the hard scrape budget is now provider+2s so per-provider timeouts can fire cleanly first - All windows tunable via SCRAPER_* env vars Addon (#64): - New debridCatalogs config flag (default true); decouples \"Debrid key set\" from \"Debrid cloud catalog enabled\" - Configure page exposes the toggle inside the Debrid Services card - Manifest skips Debrid catalogs when the flag is off Docs (#63, #66): - README \"Exposing to the Internet\" rewritten with Caddy and Cloudflare Tunnel snippets and a why-HTTPS-is-needed note - New \"Oracle Cloud / Free Tier VPS notes\" subsection covering VCN security list + iptables rules Generated with Claude Code * Mention the hosted instance in the self-hosting section Self-hosting is a choice, not a requirement. Call out magnetio.peterdsp.dev as the \"I just want it working\" path before diving into reverse proxies and Oracle VCN rules. Generated with Claude Code") | 2 weeks agoJun 18, 2026 | | [providers](https://github.com/peterdsp/Magnetio/tree/main/scraper/providers "providers") | [providers](https://github.com/peterdsp/Magnetio/tree/main/scraper/providers "providers") | [Fix scraper 429 rotation, add Debrid catalog toggle, document HTTPS/O…](https://github.com/peterdsp/Magnetio/commit/797dfb4886d061906069c51e344f4ec5ecaea99e "Fix scraper 429 rotation, add Debrid catalog toggle, document HTTPS/Oracle setup (#91) * Scraper resilience, Debrid catalog toggle, HTTPS/Oracle deploy docs Scraper (#65): - Rotate domains on HTTP 429 (previously bubbled up as an unhandled status, never tried sibling domains) - Longer cooldown when a domain is rate-limited vs hard-failed - Bumped per-provider timeout from 15s to 25s; the hard scrape budget is now provider+2s so per-provider timeouts can fire cleanly first - All windows tunable via SCRAPER_* env vars Addon (#64): - New debridCatalogs config flag (default true); decouples \"Debrid key set\" from \"Debrid cloud catalog enabled\" - Configure page exposes the toggle inside the Debrid Services card - Manifest skips Debrid catalogs when the flag is off Docs (#63, #66): - README \"Exposing to the Internet\" rewritten with Caddy and Cloudflare Tunnel snippets and a why-HTTPS-is-needed note - New \"Oracle Cloud / Free Tier VPS notes\" subsection covering VCN security list + iptables rules Generated with Claude Code * Mention the hosted instance in the self-hosting section Self-hosting is a choice, not a requirement. Call out magnetio.peterdsp.dev as the \"I just want it working\" path before diving into reverse proxies and Oracle VCN rules. Generated with Claude Code") | 2 weeks agoJun 18, 2026 | | [.dockerignore](https://github.com/peterdsp/Magnetio/blob/main/scraper/.dockerignore ".dockerignore") | [.dockerignore](https://github.com/peterdsp/Magnetio/blob/main/scraper/.dockerignore ".dockerignore") | [Initial Magnetio project scaffold](https://github.com/peterdsp/Magnetio/commit/73ad77a53b1eac7f6a4fb953a85b5936a624461b "Initial Magnetio project scaffold Add full project scaffold for Magnetio: introduces addon and scraper services, Dockerfiles, docker-compose, documentation (README.md, CODE.md) and numerous library modules and entry points. Key additions: addon server (index.js, addon.js, serverless routing), addon libs (manifest, configuration, cache, filter, sort, streamInfo, magnetHelper, logger, namedQueue, landing template, types), moch debrid integrations and static injections, scraper service (index, providers, http client, cache, logger, title helper), package.json files, .env examples and .dockerignore files. Provides Redis-backed caching, Prometheus metrics middleware, configuration parsing/presets, and health endpoints — a ready base for building the Stremio torrent aggregator with multi-debrid support.") | 4 months agoMar 14, 2026 | | [.env.example](https://github.com/peterdsp/Magnetio/blob/main/scraper/.env.example ".env.example") | [.env.example](https://github.com/peterdsp/Magnetio/blob/main/scraper/.env.example ".env.example") | [Initial Magnetio project scaffold](https://github.com/peterdsp/Magnetio/commit/73ad77a53b1eac7f6a4fb953a85b5936a624461b "Initial Magnetio project scaffold Add full project scaffold for Magnetio: introduces addon and scraper services, Dockerfiles, docker-compose, documentation (README.md, CODE.md) and numerous library modules and entry points. Key additions: addon server (index.js, addon.js, serverless routing), addon libs (manifest, configuration, cache, filter, sort, streamInfo, magnetHelper, logger, namedQueue, landing template, types), moch debrid integrations and static injections, scraper service (index, providers, http client, cache, logger, title helper), package.json files, .env examples and .dockerignore files. Provides Redis-backed caching, Prometheus metrics middleware, configuration parsing/presets, and health endpoints — a ready base for building the Stremio torrent aggregator with multi-debrid support.") | 4 months agoMar 14, 2026 | | [Dockerfile](https://github.com/peterdsp/Magnetio/blob/main/scraper/Dockerfile "Dockerfile") | [Dockerfile](https://github.com/peterdsp/Magnetio/blob/main/scraper/Dockerfile "Dockerfile") | [Initial Magnetio project scaffold](https://github.com/peterdsp/Magnetio/commit/73ad77a53b1eac7f6a4fb953a85b5936a624461b "Initial Magnetio project scaffold Add full project scaffold for Magnetio: introduces addon and scraper services, Dockerfiles, docker-compose, documentation (README.md, CODE.md) and numerous library modules and entry points. Key additions: addon server (index.js, addon.js, serverless routing), addon libs (manifest, configuration, cache, filter, sort, streamInfo, magnetHelper, logger, namedQueue, landing template, types), moch debrid integrations and static injections, scraper service (index, providers, http client, cache, logger, title helper), package.json files, .env examples and .dockerignore files. Provides Redis-backed caching, Prometheus metrics middleware, configuration parsing/presets, and health endpoints — a ready base for building the Stremio torrent aggregator with multi-debrid support.") | 4 months agoMar 14, 2026 | | [index.js](https://github.com/peterdsp/Magnetio/blob/main/scraper/index.js "index.js") | [index.js](https://github.com/peterdsp/Magnetio/blob/main/scraper/index.js "index.js") | [Add Torznab, Jackett, Prowlarr, TMDB recommendations, and marketing r…](https://github.com/peterdsp/Magnetio/commit/b00700b48713ed88978d8faf86ec852acdeab039 "Add Torznab, Jackett, Prowlarr, TMDB recommendations, and marketing redesign (#40) * Add Torznab provider and optional Jackett/Prowlarr containers Add a Torznab scraper provider that queries any Torznab-compatible endpoint (Jackett, Prowlarr, or standalone). Users configure the Torznab URL and API key in the configure page; the provider is conditionally active only when a URL is set. Add Jackett and Prowlarr as optional Docker Compose services using profiles. Start them with --profile jackett or --profile prowlarr. * Fix Torznab URL input missing dark background styling Wrap the Torznab URL input in a password-wrap div to match the styling of all other input fields on the configure page. * Redesign configure page with split layout and editorial typography Inspired by Figma community design #45. Split layout with sticky hero panel (Bitter serif headings, Space Mono body, numbered feature callouts) and scrollable form panel. Warm peach accent (#ffcfaa) replacing the previous green. Pill-shaped buttons, responsive mobile layout with collapsible hero. * Remove broken logo image from hero panel * Update README with Torznab, Jackett, and Prowlarr documentation Add Torznab provider to the provider tables, document Jackett and Prowlarr as optional Docker Compose services with profiles, add torznabUrl/torznabKey to the configuration parameters table, and update Quick Start with profile commands. * Forward client IP to Real-Debrid API calls Use AsyncLocalStorage to capture the Stremio client IP from the Express request and pass it as the `ip` parameter on every RD POST call (addMagnet, selectFiles, unrestrict/link). This lets RD associate the unrestricted download with the end user device instead of the server, preventing IP mismatch errors when streaming outside the home network. * Restyle hero panel with Figma-inspired editorial layout Update the configure page hero panel with mixed serif/italic headline typography, a 2x2 glassmorphic stats grid with real feature counts, and refined spacing. Keeps all form functionality intact. * Add similar content recommendations via TMDB API When a TMDB API key is configured, Magnetio shows a \"Similar\" catalog for every movie and series in Stremio. Recommendations are fetched from TMDB, enriched with IMDb IDs, and cached for 24 hours. New files: addon/lib/similar.js (TMDB recommendation engine) Modified: manifest.js (register similar catalogs), addon.js (route catalog handler), configuration.js (parse tmdb key), landingTemplate.js (add Recommendations section with TMDB API key field). * Redesign landing page as full marketing showcase Dark cinematic theme with gradient accents, glass-morphism cards, and animated floating orbs. Full page sections: hero with gradient headline, stats bar (22+ providers, 8 debrid, 500+ indexers), 6-card feature grid, how-it-works flow, configure form, install bar, and footer. All form IDs and JS logic preserved. * Add SEO meta tags, structured data, sitemap, and robots.txt Add comprehensive SEO to the landing page: title tag with keywords, meta description, Open Graph tags, Twitter Card, JSON-LD structured data (SoftwareApplication schema), canonical URL. Add /robots.txt and /sitemap.xml endpoints to serverless.js. Update GitHub repo: description, homepage URL, and 20 topic tags covering all key search terms (stremio, debrid, jackett, prowlarr, torznab, self-hosted, etc.). Update README with additional badges and links to magnetio.peterdsp.dev. * Address Copilot review: security, caching, encoding, and docs - Include torznab API key hash in cache keys (scraper + addon) - Validate torznab URL scheme (http/https only) to reduce SSRF risk - Redact full URL from logs, show only origin - Coerce NaN to 0 for parsed numeric torznab fields - Wrap decodeURIComponent in try/catch for malformed encoding - URL-encode torznabKey in config string builder - Use TorrentProvider.TORZNAB constant instead of hardcoded string - Update scrapeAll JSDoc for context parameter") | last monthJun 1, 2026 | | [package-lock.json](https://github.com/peterdsp/Magnetio/blob/main/scraper/package-lock.json "package-lock.json") | [package-lock.json](https://github.com/peterdsp/Magnetio/blob/main/scraper/package-lock.json "package-lock.json") | [Bump the scraper-minor-patch group across 1 directory with 2 updates (](https://github.com/peterdsp/Magnetio/commit/c5dfb24bb5a51e9ffb99bda9eeba0f25825176e7 "Bump the scraper-minor-patch group across 1 directory with 2 updates (#95) Bumps the scraper-minor-patch group with 2 updates in the /scraper directory: [axios](https://github.com/axios/axios) and [node-cron](https://github.com/node-cron/node-cron). Updates `axios` from 1.18.0 to 1.18.1 - [Release notes](https://github.com/axios/axios/releases) - [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md) - [Commits](https://github.com/axios/axios/compare/v1.18.0...v1.18.1) Updates `node-cron` from 4.4.1 to 4.5.0 - [Release notes](https://github.com/node-cron/node-cron/releases) - [Changelog](https://github.com/node-cron/node-cron/blob/main/CHANGELOG.md) - [Commits](https://github.com/node-cron/node-cron/compare/v4.4.1...v4.5.0) --- updated-dependencies: - dependency-name: axios dependency-version: 1.18.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: scraper-minor-patch - dependency-name: node-cron dependency-version: 4.5.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: scraper-minor-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>") [#…](https://github.com/peterdsp/Magnetio/pull/95) | last weekJun 25, 2026 | | [package.json](https://github.com/peterdsp/Magnetio/blob/main/scraper/package.json "package.json") | [package.json](https://github.com/peterdsp/Magnetio/blob/main/scraper/package.json "package.json") | [Bump the scraper-minor-patch group across 1 directory with 2 updates (](https://github.com/peterdsp/Magnetio/commit/c5dfb24bb5a51e9ffb99bda9eeba0f25825176e7 "Bump the scraper-minor-patch group across 1 directory with 2 updates (#95) Bumps the scraper-minor-patch group with 2 updates in the /scraper directory: [axios](https://github.com/axios/axios) and [node-cron](https://github.com/node-cron/node-cron). Updates `axios` from 1.18.0 to 1.18.1 - [Release notes](https://github.com/axios/axios/releases) - [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md) - [Commits](https://github.com/axios/axios/compare/v1.18.0...v1.18.1) Updates `node-cron` from 4.4.1 to 4.5.0 - [Release notes](https://github.com/node-cron/node-cron/releases) - [Changelog](https://github.com/node-cron/node-cron/blob/main/CHANGELOG.md) - [Commits](https://github.com/node-cron/node-cron/compare/v4.4.1...v4.5.0) --- updated-dependencies: - dependency-name: axios dependency-version: 1.18.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: scraper-minor-patch - dependency-name: node-cron dependency-version: 4.5.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: scraper-minor-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>") [#…](https://github.com/peterdsp/Magnetio/pull/95) | last weekJun 25, 2026 | | View all files | You can’t perform that action at this time.