Dry-Run First: Building a Privacy-Preserving and Idempotent Document Sorting Pipeline

In Part 1 and Part 2 of this series, we tackled the extraction and orientation challenges of raw mobile document scans: reading text with native Windows OCR in memory and detecting upside-down pages using dictionary density scoring.

However, the most dangerous phase of any local automation pipeline is the actual file movement and restructuring phase. Beginners write scripts that immediately call Move-Item or os.rename() as soon as a condition matches. In real-world enterprise environments, this causes catastrophic failure:

• Destination Collisions: If two unsequenced files receive the same target name (e.g., both get tagged as 02_Sanction_Letter.jpg), the second move silently overwrites the first, permanently destroying legal evidence.
• Midway Script Crashes: If an unhandled exception or locked file interrupts execution on file 54 of 98, your folder is left in a corrupted half-sorted state with no audit trail of what was moved.
• Outlier Pollution: Incidental photos (like a desk calculator or payment voucher) get forced into legal contracts, corrupting merged PDFs.

To eliminate these risks, we employ Dry-Run First Architecture: a zero-data-loss methodology that guarantees idempotency and safe execution.

1. Architecture: The Dry-Run Validation Gate

flowchart LR
    A["📂 98 Raw Scans & Metadata"] --> B["🧠 In-Memory JSON Simulation"]
    B --> C{"🛡️ Pre-Flight Invariant Gate
5 Assertion Rules"}
    C -->|Passed| D["📋 Copy2 with Timestamps & SHA-256 Checksum"]
    D --> E["🗄️ Read-Only Original Archive"]
    C -->|Outliers <0.20| F["⚠️ Staging: _Unsorted_Outliers/"]

    style A fill:#1e293b,stroke:#3b82f6,stroke-width:2px,color:#f8fafc
    style B fill:#1e293b,stroke:#a855f7,stroke-width:2px,color:#f8fafc
    style C fill:#1e293b,stroke:#f59e0b,stroke-width:2px,color:#f8fafc
    style D fill:#1e293b,stroke:#10b981,stroke-width:2px,color:#f8fafc
    style E fill:#064e3b,stroke:#34d399,stroke-width:2px,color:#f8fafc
    style F fill:#451a03,stroke:#f97316,stroke-width:2px,color:#f8fafc
  

flowchart LR
    A["98 Raw Scans & Metadata"] --> B["In-Memory JSON Simulation"]
    B --> C{"Pre-Flight Invariant Gate (5 Assertion Rules)"}
    C -->|Passed| D["Copy2 with Timestamps & SHA-256 Checksum"]
    D --> E["Read-Only Original Archive"]
    C -->|Outliers <0.20| F["Staging: _Unsorted_Outliers/"]

The core principle is simple: computation is cheap; file corruption is fatal. We separate execution into two completely isolated phases: manifest simulation and atomic movement.

2. Phase 1: Generating the In-Memory Execution Manifest

Instead of touching disk, the classification engine outputs an immutable JSON execution plan detailing every proposed movement:

{
  "batch_id": "20261004_LEGAL_98",
  "total_scans": 98,
  "actions": [
    {
      "source_file": "IMG_1879789263.jpg",
      "target_folder": "01_Master_Facility_Agreement",
      "target_file": "01_Facility_Agreement_Page_01.jpg",
      "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "visual_rotation": 0
    }
  ],
  "unmapped_outliers": [
    "IMG_1879789312.jpg", "IMG_1879789315.jpg", "IMG_1879789340.jpg"
  ]
}

3. Phase 2: Pre-Flight Assertion Verification

Before any folder is created or file copied, a verification gate asserts five mathematical invariants:

import os
import hashlib

def verify_sorting_manifest(manifest):
    source_set = set()
    dest_set = set()
    
    for action in manifest["actions"]:
        src = action["source_file"]
        dst = os.path.join(action["target_folder"], action["target_file"])
        
        # Invariant 1: Source file must physically exist and be readable
        if not os.path.isfile(src):
            raise FileNotFoundError(f"Missing source file: {src}")
            
        # Invariant 2: No duplicate source mappings (cannot move same file twice)
        if src in source_set:
            raise ValueError(f"Duplicate source assignment detected: {src}")
        source_set.add(src)
        
        # Invariant 3: Zero destination collisions (no overwriting)
        if dst in dest_set or os.path.exists(dst):
            raise FileExistsError(f"Collision detected at destination: {dst}")
        dest_set.add(dst)
        
    # Invariant 4: Mass balance conservation (mapped + outliers == total input)
    total_accounted = len(source_set) + len(manifest["unmapped_outliers"])
    assert total_accounted == manifest["total_scans"], "Mass balance mismatch!"
    
    print(f"Pre-flight gate passed: {len(source_set)} files verified for atomic sorting.")

4. Phase 3: Copy-Then-Verify Rather than Move

Never perform an in-place move on unbacked data. The safest pattern is Copy-Verify-Archive:

1. Copy with Metadata: Copy the source image to the target directory with its zero-padded sequential name using shutil.copy2() to preserve original timestamps.
2. Checksum Verification: Calculate SHA-256 hashes of the target file to ensure zero bytes were dropped.
3. Archive Originals: Move the raw input folder into a read-only _Original_Raw_Scans/ archive rather than deleting them.

5. Outlier Governance: Preserving Non-Document Files

In our 98-image transaction batch, three files were casual snapshots: a photo of the leather folder cover, a desk calculator showing cash settlement figures, and an office envelope. Brittle scripts force every file into a category, placing a photo of a calculator as “Page 18 of Loan Agreement”.

In our dry-run architecture, files with low confidence (< 0.20) are marked as Unmapped Outliers. They remain in an _Unsorted_Outliers/ staging folder for human inspection. Nothing is lost, and formal legal binders remain pristine.

Executive Takeaways

• Zero Data Loss Guarantee: By validating pre-flight invariants and using copy-verify-archive patterns, scripts can never overwrite documents or corrupt batches.
• Idempotent Reruns: Running the pipeline three times produces the exact same clean state without creating duplicate subfolders or broken links.
• Audit-Ready Compliance: The dry-run manifest acts as an immutable cryptographic audit record of how physical evidence was indexed.