# SmartDetect Deduplication — Android/Kotlin

Adapted from AIOStreams' `StreamDeduplicator.smartDetect` for NexStream's Kotlin `StreamDeduplicator`.

## The Problem

Filename dedup (normalize + lowercase) catches most duplicates, but cross-addon duplicates where different addons format filenames differently slip through:

- Addon A: `The.Movie.2024.1080p.BluRay.x264-FLUX.mkv` → 1.52 GB
- Addon B: `The Movie 2024 1080p BluRay x264 FLUX` → 1.48 GB

These are the SAME FILE but filename normalization may not match. Adding `infoHash` helps when available, but not all addons provide hashes.

## The Solution: Geometric Size Bucketing

Files within ~10% of each other in size are likely the same file. Geometric rounding maps size to bucket midpoints:

```kotlin
import kotlin.math.exp
import kotlin.math.ln
import kotlin.math.roundToLong

fun geometricRound(value: Long, pct: Double = 10.0): Long {
    if (value <= 0) return 0L
    val logStep = ln(1.0 + pct / 100.0)
    val bucketIndex = (ln(value.toDouble()) / logStep).roundToLong()
    return exp(bucketIndex * logStep).roundToLong()
}
```

Examples at 10%:
- 1_500_000_000 → ~1_520_000_000
- 1_520_000_000 → ~1_520_000_000 (same bucket)
- 1_700_000_000 → ~1_720_000_000 (different bucket)

## Critical: Run as a SECOND PASS, Not Union-Forming

**NEVER add SmartDetect as a union-find key.** If you do, streams with different content but similar sizes (e.g., DV 20GB vs SDR 20GB from the same movie) get falsely merged.

### Why union-find fails

Union-find merges streams that share ANY key. If stream A and stream B share `sd:rounded_20GB:4K`:
- Stream A: "Movie.2160p.Dolby.Vision.HEVC" → DV encode, 20GB
- Stream B: "Movie.2160p.HEVC" → SDR encode, 20GB

They get union'd even though they're genuinely different files.

### Correct: second-pass grouping

After primary dedup (filename + infoHash union-find), group survivors by SmartDetect fingerprint. Within each group, keep only the best using standard tie-breaking:

```kotlin
fun deduplicate(streams, config, addonPriorities, useSmartDetect = false): List<Stream> {
    // ... primary union-find dedup ...

    // Second pass: SmartDetect grouping (not union-forming)
    if (useSmartDetect && result.size > 1) {
        result = smartDetectSecondPass(result, addonPriorities)
    }
    return result
}
```

## Fingerprint to Prevent False Merges

A fingerprint from the stream title prevents merging genuinely different files at the same size:

```kotlin
private fun buildSmartDetectFingerprint(stream: Stream): String {
    val text = (stream.name ?: stream.title ?: "").lowercase()
    val parts = mutableListOf<String>()

    // Resolution
    when {
        "2160" in text || "4k" in text -> parts.add("4K")
        "1080" in text || "fhd" in text -> parts.add("1080p")
        "720" in text || "hd" in text -> parts.add("720p")
    }

    // HDR
    when {
        "dolby" in text || "dv" in text -> parts.add("DV")
        "hdr10+" in text -> parts.add("HDR+")
        "hdr10" in text || "hdr" in text -> parts.add("HDR")
    }

    // Codec
    when {
        "av1" in text -> parts.add("AV1")
        "hevc" in text || "x265" in text -> parts.add("HEVC")
        "avc" in text || "x264" in text -> parts.add("AVC")
    }

    // Name tail (last 8 chars of normalized name) for release-group distinction
    val norm = normalizeFilename(stream.name ?: stream.title ?: "")
    if (norm.length >= 8) parts.add(norm.takeLast(8))

    return if (parts.isEmpty()) "" else ":${parts.joinToString(".")}"
}
```

The fingerprint produces keys like: `sd:1520000000:4K.DV.HEVC.x265flux`

Two streams with the SAME key are almost certainly the same file. Two streams with different keys (DV vs SDR at same size) are kept separate.

## Default: Off

SmartDetect defaults to `useSmartDetect = false`. Enable it in `SourceRanker` with `useSmartDetect = true` since SourceRanker provides reliable size data from behavior hints.

## Test-Driven Validation

The test suite should verify that SmartDetect does NOT false-merge:
1. DV vs SDR at same size → kept separate (different HDR fingerprint)
2. AV1 vs H264 at same size → kept separate (different codec fingerprint)
3. GROUPA vs GROUPB at same size/resolution → kept separate (different name-tail)
4. Same file, same size, different addon → merged ✓