Deduplication

Purpose and authority boundary

Deduplication consumes raw Search/import sightings and preserves their lineage while identifying exact duplicate components and ambiguous review pairs. Automatic cluster formation is limited to accepted stable-identifier evidence. Fuzzy title, source-specific ID, and no-ID evidence can request review but cannot silently become scientific identity.

A representative is a projection over preserved members and evidence. It does not delete raw sightings. A review request is not a decision, and a decision is not a published corpus snapshot; those authority steps remain explicit.

Analysis model

SearchTrace / SearchImportTrace
  → DedupCandidateRecord (all source bindings retained)
  → pair evidence
       exact stable ID → automatic union-find edge
       fuzzy title ≥ 0.95 → human-review edge
       source-specific/no ID → human-review evidence
  → DedupCluster + DedupReviewCandidate + unresolved candidates
  → deterministic representative projection

Exact edges are transitive, but every direct edge remains recorded. The representative selector ranks metadata completeness, stable identifiers, recorded provider priority, DOI tie-break, normalized primary identifier, and deterministic candidate id. It fills missing fields and unions identifiers; it does not overwrite populated representative data silently.

Key types and services

• DeduplicationService builds candidates/evidence, exact clusters, fuzzy review pairs, representatives, warnings, errors, and nonclaims.
• DeduplicationResult, DedupCluster, DedupEvidence, and DedupReviewCandidate retain the complete analysis surface.
• DeduplicationRehydrator returns VerifiedDeduplicationResult only after complete semantic/order validation.
• DeduplicationAuthorityDigests creates verified result, candidate, evidence, and exact review-target digests.
• DeduplicationAuthorityPolicy verifies allowed human actor/role pairs, actions, reason codes, rationale requirements, issuer, and policy digest.
• DeduplicationReviewCommand binds the exact generation, active decisions, result, snapshot, target, policy, action, actor, and optional supersession into a stable request digest/idempotency key.
• DeduplicationDecision produces append-only verified human decisions for merge, keep-separate, and mark-unresolved.

Source: NexusScholar.Deduplication. Contracts: ADR 0012, ADR 0028, and ADR 0029.

Human decision transition

• merge: unions the groups/unresolved entries containing the verified pair, preserves all members/evidence, and elects the representative deterministically.
• keep-separate: preserves membership and creates an active separation constraint.
• mark-unresolved: preserves membership and records an active unresolved target that blocks a final-corpus claim.

The request digest excludes time and random IDs and deterministically derives request/decision IDs. The committed decision digest includes transaction time. Exact retries return the stored transition; a new action must explicitly supersede the current decision by id and digest. An accepted transition is published atomically with a successor corpus snapshot, invalidation, and provenance in ResearchWorkspace.

Authority digests and stale-state defense

The canonical result digest covers policy, ordered source bindings, all raw candidates/sightings, clusters, representatives, evidence, unresolved candidates, review pairs, warnings, errors, and nonclaims. Separate candidate/evidence/review-target digests prevent an unchanged id from hiding changed content. Commands additionally bind the predecessor snapshot and active decision-set digest, so an old browser/CLI action cannot apply to newer authority.

Failure and refusal behavior

The module rejects title-only automatic merge, runtime/path identity, fuzzy auto-merge, malformed or noncanonical results, missing source bindings, incomplete representative metadata, stale targets/policies/snapshots/generations, unauthorized actors, unapproved actions/reasons, implicit overwrite, missing or cross-target supersession, cycles, duplicate active decisions, and merge across an active keep-separate constraint.

Tests, extension points, and nonclaims

DeduplicationServiceTests, decision tests, command tests, and conformance tests cover clustering, representation, authority, stale state, and human transitions. Fixtures are under deduplication and executable review.

New matching policies must preserve raw evidence, record thresholds/version, and classify automatic versus review-only authority. This package provides no learned similarity model, generic cross-domain decision framework, manual representative override, final corpus publication, UI/database/cloud behavior, or broad PHP compatibility.