Repository navigation
Phase 25: Advanced De-identification - #47
Conversation
Phase 25: Advanced De-identification - Implementation decisions documented - Phase boundary established
Phase 25: Advanced De-identification - Tesseract C API surface documented (capi.h functions, PSM modes) - Leptonica dependency chain verified (libpng + zlib required) - Zig cross-compilation risk flagged for macOS targets - VR-based UID reference walking pattern documented - Existing Phase 14 integration points catalogued - Common pitfalls identified (16-bit conversion, MONOCHROME1 inversion, memory management)
Phase 25: Advanced De-identification - 4 plans in 3 waves - 2 parallel (Wave 1), 1 sequential (Wave 2), 1 sequential (Wave 3) - Ready for execution
Address 5 blockers and 3 warnings: - Clarify ROADMAP should-haves as deferred per CONTEXT.md - Add zig build verification command to Plan 25-02 Task 2 - Wire tesseract_wrapper.h into sharpdicom_codecs.c - Implement compressed pixel data decompression via CodecRegistry - Add 25-02 to Plan 25-04 depends_on - Split Plan 25-03 Task 2 into 2a (OcrScanner) and 2b (pipeline) - Add multi-valued UID backslash splitting to Plan 25-01 - Establish definitive pipeline order across Plans 25-01 and 25-03
- UidReferenceWalker recursively walks all VR=UI elements at unlimited depth - Handles multi-valued UIDs (backslash-separated) per component - Preserves standard DICOM UIDs (1.2.840.10008.*) - UidRemapResult provides traversal statistics and remapped tag list
- tesseract_wrapper.h: API declarations for 10 wrapper functions - tesseract_wrapper.c: full Tesseract C API wrapper and stub mode - TessDetectionResult struct for word-level OCR results - Stub mode when SHARPDICOM_WITH_TESSERACT not defined - Follows existing wrapper patterns (jpeg_wrapper, jls_wrapper)
…line - Add walkAllUidReferences parameter to DicomDeidentifier constructor - Pipeline order: ProcessDataset -> date shift -> UID reference walk -> markers - Add WithUidReferenceWalking() to DicomDeidentifierBuilder for opt-in - Add UidReferencesRemapped to DeidentificationSummary for tracking - Fully backward-compatible: existing callers unaffected
Tasks completed: 2/2 - UidReferenceWalker and UidRemapResult types - DicomDeidentifier and Builder pipeline integration SUMMARY: .planning/phases/25-advanced-deidentification/25-01-SUMMARY.md
…aged P/Invoke layer - build.zig: add tesseract_wrapper.c to all build targets (stub mode) - sharpdicom_codecs.h: add SHARPDICOM_HAS_TESSERACT feature constant - sharpdicom_codecs.c: include tesseract_wrapper.h, add Tesseract feature flag - TesseractNativeMethods.cs: P/Invoke for 10 functions (LibraryImport + DllImport) - TesseractHandle.cs: SafeHandle for TessBaseAPI lifecycle management - NativeMethods.cs: add Tesseract flag to NativeFeatures enum - TessDetection struct for managed detection results
Tasks completed: 2/2 - Task 1: Native Tesseract C wrapper (tesseract_wrapper.h/c) - Task 2: Zig build integration, sharpdicom_codecs wiring, managed P/Invoke layer SUMMARY: .planning/phases/25-advanced-deidentification/25-02-SUMMARY.md
- OcrScanModality flags enum for risk-based modality filtering - OcrScannerOptions with confidence thresholds, edge zones, allow/deny lists - DefaultNonPhiAllowlist with orientation markers, units, medical abbreviations - OcrDetection readonly record struct with text, confidence, bounding box - OcrScanResult with filtered detections and ToRedactionRegions() converter
- OcrScanner sealed class with Tesseract P/Invoke interop - Handles 8-bit, 16-bit, and RGB pixel data with MONOCHROME1 inversion - 16-bit windowed to 8-bit using Window Center/Width or full-range normalisation - RGB converted to grayscale via ITU-R BT.601 luminance formula - Compressed pixel data decompressed via CodecRegistry when DecompressForOcr is true - Edge/center dual-threshold confidence filtering - Allow/deny list filtering for non-PHI text (orientation markers, units) - Fail-fast with clear exception when Tesseract native library unavailable - Multi-frame scanning processes every frame
- DicomDeidentifierBuilder.WithOcrScanner() enables OCR in the pipeline - DicomDeidentifier accepts OcrScannerOptions, lazy-creates OcrScanner - Pipeline order: OCR scan -> ProcessDataset -> date shifting -> UID walking -> markers - OCR runs BEFORE primary de-id to preserve metadata needed for scanning - DeidentificationSummary extended with OCR statistics - OcrScanner disposed in DicomDeidentifier.Dispose() - Full backward compatibility: callers not using WithOcrScanner() are unaffected
Tasks completed: 3/3 - OcrScannerOptions and OcrScanResult types - OcrScanner standalone implementation - DicomDeidentifier pipeline integration for OCR scanning SUMMARY: .planning/phases/25-advanced-deidentification/25-03-SUMMARY.md
- 18 tests covering core remapping, sequence traversal, consistency, RT/SR reference patterns, and result type verification - Tests verify recursive VR=UI traversal at arbitrary depth (10 levels) - Multi-valued UID per-component remapping validated - Standard DICOM UIDs (1.2.840.10008.*) proven never remapped
- 14 OcrScannerOptions tests: defaults, allowlist content, case sensitivity - 10 OcrScanner tests: constructor fail-fast, result conversion, filtering - 9 integration tests: pipeline with/without UID walking, OCR unavailability, multi-valued UIDs, standard UID preservation, cross-file consistency - All tests work without Tesseract native library installed
Tasks completed: 2/2 - UidReferenceWalker unit tests (18 tests) - OCR scanner, options, and integration tests (32 tests) SUMMARY: .planning/phases/25-advanced-deidentification/25-04-SUMMARY.md
![recurseml[bot]](https://avatars.githubusercontent.com/in/1054195?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review by RecurseML
🔍 Review performed on 17d7aa8..76a9b09
✨ No bugs found, your code is sparkling clean
✅ Files analyzed, no issues (33)
• .planning/ROADMAP.md
• .planning/STATE.md
• .planning/phases/25-advanced-deidentification/25-01-PLAN.md
• .planning/phases/25-advanced-deidentification/25-01-SUMMARY.md
• .planning/phases/25-advanced-deidentification/25-02-PLAN.md
• .planning/phases/25-advanced-deidentification/25-02-SUMMARY.md
• .planning/phases/25-advanced-deidentification/25-03-PLAN.md
• .planning/phases/25-advanced-deidentification/25-03-SUMMARY.md
• .planning/phases/25-advanced-deidentification/25-04-PLAN.md
• .planning/phases/25-advanced-deidentification/25-04-SUMMARY.md
• .planning/phases/25-advanced-deidentification/25-CONTEXT.md
• .planning/phases/25-advanced-deidentification/25-RESEARCH.md
• .planning/phases/25-advanced-deidentification/25-VERIFICATION.md
• native/build.zig
• native/src/sharpdicom_codecs.c
• native/src/sharpdicom_codecs.h
• native/src/tesseract_wrapper.c
• native/src/tesseract_wrapper.h
• src/SharpDicom.Codecs/Interop/NativeMethods.cs
• src/SharpDicom.Codecs/Interop/TesseractHandle.cs
• src/SharpDicom.Codecs/Interop/TesseractNativeMethods.cs
• src/SharpDicom/Deidentification/DeidentificationResult.cs
• src/SharpDicom/Deidentification/DicomDeidentifier.cs
• src/SharpDicom/Deidentification/DicomDeidentifierBuilder.cs
• src/SharpDicom/Deidentification/OcrScanResult.cs
• src/SharpDicom/Deidentification/OcrScanner.cs
• src/SharpDicom/Deidentification/OcrScannerOptions.cs
• src/SharpDicom/Deidentification/UidReferenceWalker.cs
• src/SharpDicom/Deidentification/UidRemapResult.cs
• tests/SharpDicom.Tests/Deidentification/AdvancedDeidentificationIntegrationTests.cs
• tests/SharpDicom.Tests/Deidentification/OcrScannerOptionsTests.cs
• tests/SharpDicom.Tests/Deidentification/OcrScannerTests.cs
• tests/SharpDicom.Tests/Deidentification/UidReferenceWalkerTests.cs
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #47 +/- ##
==========================================
- Coverage 65.76% 64.72% -1.04%
==========================================
Files 231 236 +5
Lines 18658 19187 +529
Branches 3494 3602 +108
==========================================
+ Hits 12270 12419 +149
- Misses 5438 5815 +377
- Partials 950 953 +3 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
2 issues found across 33 files
Prompt for AI agents (all issues)
Check if these issues are valid — if so, understand the root cause of each and fix them.
<file name="src/SharpDicom/Deidentification/OcrScanner.cs">
<violation number="1" location="src/SharpDicom/Deidentification/OcrScanner.cs:519">
P1: `double.TryParse` without `CultureInfo.InvariantCulture` will misparse DICOM Window Center/Width values on non-US locales, causing incorrect 16-bit to 8-bit windowing. DICOM DS values always use `.` as the decimal separator regardless of locale.</violation>
</file>
<file name="src/SharpDicom.Codecs/Interop/TesseractNativeMethods.cs">
<violation number="1" location="src/SharpDicom.Codecs/Interop/TesseractNativeMethods.cs:171">
P2: The netstandard/legacy P/Invoke for `tess_init` marshals strings as ANSI (`LPStr`), which will corrupt non‑ASCII tessdata paths on Windows and is inconsistent with the NET7 UTF‑8 marshalling. Use UTF‑8 marshalling (e.g., `LPUTF8Str` or manual UTF‑8 byte conversion) to avoid initialization failures.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
Phase 26: Migration Tooling - Implementation decisions documented - Phase boundary established
- Use CultureInfo.InvariantCulture for double.TryParse of Window Center/Width DS values (DICOM always uses '.' decimal separator) - Change tess_init P/Invoke from LPStr (ANSI) to LPUTF8Str (value 48) for non-ASCII tessdata paths, matching the NET7+ LibraryImport
Summary
Test plan
High-level PR Summary
This PR delivers Phase 25: Advanced De-identification, adding OCR-based burned-in PHI detection and comprehensive UID reference walking to the DICOM de-identification pipeline. The
UidReferenceWalkerrecursively remaps allVR=UIelements across sequences at unlimited depth, handling multi-valued UIDs per-component while preserving standard DICOM UIDs. A native Tesseract C wrapper with stub compilation mode integrates via P/Invoke (LibraryImportfor NET7+,DllImportfor netstandard2.0), exposing OCR text detection capabilities. TheOcrScannerclass detects burned-in text in 8-bit, 16-bit windowed, RGB, and MONOCHROME1-inverted pixel data, decompresses encapsulated frames viaCodecRegistry, and applies dual-threshold confidence filtering with allow/deny lists for non-PHI text (orientation markers, units, abbreviations). Both features integrate intoDicomDeidentifieras opt-in builder methods (WithUidReferenceWalking(),WithOcrScanner()), with pipeline order: OCR scan → primary de-identification → date shifting → UID reference walking → markers. 50 new tests across 4 test files bring the total to 2263 tests (2209 pass, 54 skipped, 0 failures), with comprehensive coverage of recursive traversal, OCR configuration, pixel format handling, and cross-file UID consistency.⏱️ Estimated Review Time: 3+ hours
💡 Review Order Suggestion
.planning/ROADMAP.md.planning/STATE.md.planning/phases/25-advanced-deidentification/25-01-PLAN.md.planning/phases/25-advanced-deidentification/25-02-PLAN.md.planning/phases/25-advanced-deidentification/25-03-PLAN.md.planning/phases/25-advanced-deidentification/25-04-PLAN.md.planning/phases/25-advanced-deidentification/25-01-SUMMARY.md.planning/phases/25-advanced-deidentification/25-02-SUMMARY.md.planning/phases/25-advanced-deidentification/25-03-SUMMARY.md.planning/phases/25-advanced-deidentification/25-04-SUMMARY.md.planning/phases/25-advanced-deidentification/25-CONTEXT.md.planning/phases/25-advanced-deidentification/25-RESEARCH.md.planning/phases/25-advanced-deidentification/25-VERIFICATION.mdnative/src/tesseract_wrapper.hnative/src/tesseract_wrapper.cnative/build.zignative/src/sharpdicom_codecs.cnative/src/sharpdicom_codecs.hsrc/SharpDicom.Codecs/Interop/TesseractNativeMethods.cssrc/SharpDicom.Codecs/Interop/TesseractHandle.cssrc/SharpDicom.Codecs/Interop/NativeMethods.cssrc/SharpDicom/Deidentification/UidRemapResult.cssrc/SharpDicom/Deidentification/UidReferenceWalker.cssrc/SharpDicom/Deidentification/OcrScannerOptions.cssrc/SharpDicom/Deidentification/OcrScanResult.cssrc/SharpDicom/Deidentification/OcrScanner.cssrc/SharpDicom/Deidentification/DeidentificationResult.cssrc/SharpDicom/Deidentification/DicomDeidentifierBuilder.cssrc/SharpDicom/Deidentification/DicomDeidentifier.cstests/SharpDicom.Tests/Deidentification/UidReferenceWalkerTests.cstests/SharpDicom.Tests/Deidentification/OcrScannerOptionsTests.cstests/SharpDicom.Tests/Deidentification/OcrScannerTests.cstests/SharpDicom.Tests/Deidentification/AdvancedDeidentificationIntegrationTests.csSummary by cubic
Adds OCR-based burned-in PHI detection and a recursive UID reference walker to the de-identification pipeline to improve PHI coverage and keep cross-object UID references consistent.
New Features
Migration
Written for commit 5836366. Summary will update on new commits.