feat: standardize & normalize dataset formats

[madsysharma]

Standardize and normalize dataset formats

Closes #70

Summary

Adds a non-destructive dataset standardization pipeline that normalizes every sample into one consistent format: image dimensions, file format, channel layout, mask encoding, and metadata/labeling conventions, before training.
It is the normalization counterpart to the existing auditing script dataset_quality.py and follows the same conventions (root-level script, JSON report in dataset_reports/, unit tests in tests/unit/).

While building this against the real codebase I found a data-integrity bug in the shipped manifest, which this PR also fixes (details below).

Problem

The raw TCGA data and data_mask.csv are not uniform:

• Mixed image properties: source TIFs can be grayscale / RGB / RGBA, 8/16-bit, and are not guaranteed to share an exact size.
• Inconsistent mask encoding: masks may carry stray channels or non-binary values; training only needs a clean binary tumor region.
• Broken patient_id metadata: in data_mask.csv the patient_id column is misaligned: the same id is repeated across slices that actually belong to different patients. 3,894 of 3,929 rows have a patient_id that does not match the patient folder in their image_path. A patient-grouped train/val/test split keyed on that column would leak the same patient across splits and inflate reported metrics. The folder name in image_path is the ground-truth TCGA id, so the pipeline re-derives patient_id from it.

Changes included

File	Change
dataset_standardize.py	New standardization pipeline (manifest + pixel stages)
tests/unit/test_dataset_standardize.py	12 unit tests
PREPROCESSING.md	Full pipeline documentation
README.md	Pointer to the pipeline in the Dataset section
.gitignore	Ignore the generated dataset_standardized/ image dir

Running python dataset_standardize.py generates two artifacts (kept out of this diff; the image dir is git-ignored):

• data_mask_standardized.csv: canonical manifest with corrected metadata
• dataset_reports/standardization_report.json: run summary & consistency check

How it maps to the acceptance criteria

Acceptance criterion	How it's met
Consistent image resolution and format	Every image -> 256×256, 3-channel RGB, uint8, written as lossless PNG. A consistency_check in the report asserts uniform resolution / channels / dtype across the processed set.
Mask annotations standardized	Masks -> single-channel, resized with nearest-neighbour (no made-up boundary values), binarized to {0, 255}. The 0/1 mask label is re-derived from actual mask content, flagging mislabeled rows.
Preprocessing pipeline documented	PREPROCESSING.md documents every stage, the canonical format table, the manifest schema, usage and a training example.

Pipeline overview

Stage 1: Manifest standardization (no images required, works in a fresh clone): re-derives canonical patient_id, parses a numeric slice index, normalizes path separators, preserves the source label as original_mask, and emits a fixed column order.

Stage 2: Pixel standardization (for rows whose files exist): standardizes image & mask pixels, re-derives the label from the mask, and writes standardized copies to dataset_standardized/, mirroring the TCGA_*/ layout. Missing-file rows are skipped and counted, so the run succeeds with only the CSV present.

Format:

Property	Value
Image	256×256, RGB, uint8, PNG
Mask	256×256, 1-channel, {0,255}, nearest-neighbour resize
patient_id	derived from image folder
mask label	re-derived from mask content

Pixel-value normalization (/255 and mean-std) is intentionally left at model-feed time to match app.py's existing preprocessing, so the standardized data stays reusable across both the classification and segmentation stages. The original image_path/mask_path columns are preserved so existing code keeps working; training can switch to the new std_* columns once data is generated.

Usage

python dataset_standardize.py

Testing

Source-first: cloned the repo and verified everything against the real data_mask.csv and the actual TCGA_* folder/filename conventions.

PS C:\Users\madhu\Documents\NeuroVision> pytest tests/unit/test_dataset_standardize.py -v
================================================= test session starts =================================================
platform win32 -- Python 3.11.7, pytest-9.0.3, pluggy-1.6.0 -- C:\Program Files\Python311\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\madhu\Documents\NeuroVision
configfile: pytest.ini
plugins: anyio-4.13.0, deepeval-4.0.5, Faker-40.19.1, langsmith-0.8.8, asyncio-1.4.0, cov-7.1.0, mock-3.15.1, repeat-0.9.4, rerunfailures-16.3, xdist-3.8.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 12 items

tests/unit/test_dataset_standardize.py::test_derive_patient_id_uses_folder PASSED                                [  8%]
tests/unit/test_dataset_standardize.py::test_derive_slice_index PASSED                                           [ 16%]
tests/unit/test_dataset_standardize.py::test_standardize_image_grayscale_to_rgb PASSED                           [ 25%]
tests/unit/test_dataset_standardize.py::test_standardize_image_rgba_and_odd_size PASSED                          [ 33%]
tests/unit/test_dataset_standardize.py::test_standardize_image_uint16_is_scaled_to_uint8 PASSED                  [ 41%]
tests/unit/test_dataset_standardize.py::test_standardize_mask_is_binary_single_channel PASSED                    [ 50%]
tests/unit/test_dataset_standardize.py::test_standardize_mask_nearest_neighbour_no_intermediate_values PASSED    [ 58%]
tests/unit/test_dataset_standardize.py::test_label_from_mask PASSED                                              [ 66%]
tests/unit/test_dataset_standardize.py::test_standardize_manifest_fixes_patient_id_and_columns PASSED            [ 75%]
tests/unit/test_dataset_standardize.py::test_standardize_manifest_missing_column_raises PASSED                   [ 83%]
tests/unit/test_dataset_standardize.py::test_process_pixels_end_to_end PASSED                                    [ 91%]
tests/unit/test_dataset_standardize.py::test_process_pixels_handles_missing_files PASSED                         [100%]Running teardown with pytest sessionfinish...


================================================= 12 passed in 0.85s ==================================================
PS C:\Users\madhu\Documents\NeuroVision> pytest tests/unit/test_dataset_quality.py -v
================================================= test session starts =================================================
platform win32 -- Python 3.11.7, pytest-9.0.3, pluggy-1.6.0 -- C:\Program Files\Python311\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\madhu\Documents\NeuroVision
configfile: pytest.ini
plugins: anyio-4.13.0, deepeval-4.0.5, Faker-40.19.1, langsmith-0.8.8, asyncio-1.4.0, cov-7.1.0, mock-3.15.1, repeat-0.9.4, rerunfailures-16.3, xdist-3.8.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 5 items

tests/unit/test_dataset_quality.py::test_generate_statistics_counts PASSED                                       [ 20%]
tests/unit/test_dataset_quality.py::test_is_corrupted_flags_missing_and_blank PASSED                             [ 40%]
tests/unit/test_dataset_quality.py::test_filter_corrupted_respects_missing_policy PASSED                         [ 60%]
tests/unit/test_dataset_quality.py::test_class_imbalance_report_severe_case PASSED                               [ 80%]
tests/unit/test_dataset_quality.py::test_plot_distribution_creates_file PASSED                                   [100%]Running teardown with pytest sessionfinish...


================================================== 5 passed in 1.20s ==================================================

• git apply --check passes on a fresh clone; tests re-run green after applying.
• Real run on data_mask.csv: detected 110 unique patients (matches the README), corrected 3,894 misaligned patient_id rows, parsed all 3,929 slice indices.
• End-to-end pixel run on synthetic patient folders with mixed input sizes (200×200, 256×256, 128×180): all outputs uniform at 256×256×3 uint8, masks binary {0,255}, and mislabeled rows auto-corrected from mask content.

Tests cover the metadata helpers, image standardization (grayscale/RGBA/odd-size + uint16 to uint8 scaling), the nearest neighbour mask guarantee, manifest correction, missing-file handling, and a full end-to-end pixel run.

Notes for reviewer

• Non-destructive: originals are never modified; standardized copies and a corrected manifest are written alongside them.
• dataset_standardized/ is git-ignored, consistent with the existing TCGA*/ rule: dataset images are never committed.

PR checklist

• Code runs correctly and is tested locally
• Existing tests still pass
• Documentation added (PREPROCESSING.md and README pointer)
• Follows PEP 8; modular, commented functions
• No dataset images committed (output dir git-ignored)

[madsysharma]

Hi @Brijeshthummar02 , please review this PR. Thanks.

[madsysharma]

Hi @Brijeshthummar02 , please review this PR. Thanks.

[madsysharma]

Hi @Brijeshthummar02 , please review this PR. Thank you.