Fulmine-Labs-Mini-PACS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "76c0f56a",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Fulmine LABS mini-PACs\n",
    "\n",
    "## Overview\n",
    "\n",
    "Fulmine Labs will use medical images for various quality/testing related, machine learning (ML) initiatives. \n",
    "The best practice for managing this data is to use Digital Imaging and Communications in Medicine (DICOM) standard compliant images with a PACS-like system.\n",
    "\n",
    "The code in this project implements and tests a basic PACS with the following architecture:\n",
    "\n",
    "```\n",
    "\n",
    "[ Orthanc Repository (Open Source component) ]\n",
    "       |\n",
    "       | (DICOM Images) <----------------------------------------->  [ OHIF Viewer (Open Source component) ]\n",
    "       v\n",
    "[ Fulmine-Labs-Mini-PACS - Data Setup Script ]      \n",
    "       |\t\t\t\t\t\t\t\t\n",
    "       | (Metadata and generated images)   \n",
    "       |                                          \n",
    "[ SQLite Database ]\t\t\t\t\t\t\t\n",
    "       |\t\t\t\t\t\t\t\t\n",
    "       | (API Requests)\t\t\t\t\t\t \n",
    "       v\n",
    "[ Flask Application ]\n",
    "       |\n",
    "       | (HTTP Requests for Data)\n",
    "       v\n",
    "[ Client (Pytest, Browser) ]\n",
    "       |\n",
    "       | (Model Training Data)  [ Data Enhancements ]\n",
    "       v\n",
    "[ Fulmine-Labs-Mini-PACS - Data Setup Script ]\n",
    "       |\n",
    "       | (Data Enhancements)\n",
    "       v\n",
    "[ Anomaly Detection Model Training ]\n",
    "\n",
    "\n",
    "```\n",
    "\n",
    "The data setup script will traverse all folders in a specified location, identify DICOM images and if they have appropriate Window Center and Width DICOM header information, will convert them to PNG files at another specified location and add the related metadata to an SQLite database.\r\n",
    "\r\n",
    "The database maintains the Patient -> Study -> Series -> Image relationship, as well as tracking the output image file names and parameters used in their creation, allowing PACS-like SQL queries to be constructed.\r\n",
    "\r\n",
    "Currently supported endpoints (usually at http://127.0.0.1:5000) are:\r\n",
    "\r\n",
    "'/' - welcome message\r\n",
    "'/patients/<patient_id>' - get patient information\r\n",
    "'/studies/<study_id>' - get study information\r\n",
    "'/series/<series_id>' - get series information\r\n",
    "'/images/<image_id>' - get image information\r\n",
    "'/patients/<patient_id>/studies' - get studies for a patient\r\n",
    "'/patients/<patient_id>/studycount' - get study count for a patient\r\n",
    "'/patients/<patient_id>/seriescount' - get series count for a patient\r\n",
    "'/patients/<patient_id>/imagecount' - get image count for a patient\r\n",
    "'/patients/<patient_id>/counts' - get all counts for a patient\r\n",
    "'/patients/count' - get total patient count\r\n",
    "'/studies/count' - get total studies count\r\n",
    "'/series/count' - get total series count\r\n",
    "'/images/count' - get total images count\r\n",
    "'/imageinfo/' - get image info by providing the file name\r\n",
    "Once PNG images have been generated from the DICOM images, these are used as the basis of the 'valid' class in an ML image classifier. To reduce overfitting, additional images are generated and added to the valid class. These include:\r\n",
    "\r\n",
    "The same images with random window centers and widths\r\n",
    "The same images with light random blurring to simulate pixel interpolation or compression\r\n",
    "The same images with flips and rotations\r\n",
    "The same images zoomed in and out, also with random window centers and widths\r\n",
    "The 'invalid' class will be comprised of, for example:\r\n",
    "\r\n",
    "Non-medical images selected from the Kaggle 'real and fake' dataset\r\n",
    "The same valid images as above with simulated error/message boxes in order to help to detect anomalous conditions\r\n",
    "Some custom anomalous images, including AI-generated medical images\r\n",
    "All of the images above will be distributed randomly between training, validation and testing folders in order to train and test the model. In addition, in order to test how well the model recognizes previously unseen medical images of the same type, some custom images will be selected from Google searches and used only for testing.\r\n",
    "\r\n",
    "Overall the\n",
    "\n",
    "``` folder structure looks like this:\r\n",
    "\r\n",
    "Orthanc (DICOM images)\r\n",
    "  ├── subfolders\r\n",
    "\r\n",
    "training (PNG)\r\n",
    "  ├──train\r\n",
    "  ├──validate\r\n",
    "  ├──test\r\n",
    "\r\n",
    "Kaggle_real_and_fake_images (PNG)\r\n",
    "  ├── subfolders\r\n",
    "\r\n",
    "Custom_invalid (MIX)\r\n",
    "  ├── subfolders\r\n",
    "\r\n",
    "Custom_test_valid\r\n",
    "\r\n",
    "training_images\r\n",
    "  ├──train\r\n",
    "  │\t\t├── valid\r\n",
    "  │\t\t│  original training\r\n",
    "  │\t\t│   ├── blurred\r\n",
    "  │\t\t│   └── window_leveled\r\n",
    "  │\t\t│   └── rotate_and_flip\r\n",
    "  │\t\t│   └── zoomed\r\n",
    "  |             |     └── window_leveled\r\n",
    "  │\t\t└── invalid/\r\n",
    "  │   \t\t \t├── Kaggle_real_and_fake_images\r\n",
    "  │    \t \t\t├── copied from Custom_invalid\r\n",
    "  ├──validate\r\n",
    "  │\t\t├── valid\r\n",
    "  │\t\t│  original validate\r\n",
    "  │\t\t│   ├── blurred\r\n",
    "  │\t\t│   └── window_leveled\r\n",
    "  │\t\t│   └── rotate_and_flip\r\n",
    "  │\t\t│   └── zoomed\r\n",
    "  |             |     └── window_leveled\r\n",
    "  │\t\t└── invalid/\r\n",
    "  │   \t\t \t├── Kaggle_real_and_fake_images\r\n",
    "  │    \t \t\t├── copied from Custom_invalid\r\n",
    "  ├──test\r\n",
    " \t├── valid\r\n",
    " \t│  original test\r\n",
    "\t\t├── copied from Custom_test_valid\r\n",
    " \t└── invalid/\r\n",
    "\n",
    "```\n",
    "     \t \t├── Kaggle_real_and_fake_images\r\n",
    "      \t \t├── copied from Custom_invalid\r\n",
    "\r\n",
    "Once the data is prepared, the classifier model training is initiated. The model is saved and reloaded and used to test those images seected for testing, producing metrics on Accuracy, Precision, Recall and F1 score.\n",
    "\n",
    "\n",
    "## Author\n",
    "Duncan Henderson\n",
    "Fulmine Labs LLC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f1476f8f-4aec-4256-ac40-f7ef97a92337",
   "metadata": {},
   "outputs": [],
   "source": [
    "# For the image generation pipeline\n",
    "import os\n",
    "import random\n",
    "from random import sample\n",
    "import pydicom\n",
    "import sqlite3\n",
    "import shutil \n",
    "import logging\n",
    "from datetime import datetime\n",
    "import numpy as np\n",
    "from PIL import Image, ImageDraw, ImageFont, ImageOps\n",
    "import textwrap\n",
    "from scipy.ndimage import gaussian_filter, rotate\n",
    "from PIL import ImageFilter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f7e2db58-a816-4824-a0ae-afbc18c614fb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From C:\\Users\\Duncan\\anaconda3\\envs\\OHIF\\Lib\\site-packages\\keras\\src\\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# For training and testing the model\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras.layers import Conv2D, MaxPooling2D, UpSampling2D, Input\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
    "from tensorflow.keras.layers import Flatten, Dense\n",
    "from tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Flatten, Dense\n",
    "from keras.utils import to_categorical\n",
    "import scipy\n",
    "from matplotlib import pyplot as plt\n",
    "from keras.preprocessing.image import load_img, img_to_array\n",
    "from tensorflow.keras.models import load_model\n",
    "from tensorflow.keras.preprocessing.image import img_to_array, load_img\n",
    "from tensorflow.keras.layers import Dropout\n",
    "from keras.callbacks import EarlyStopping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "27aa2457-15bd-4d94-96ae-2ab7e7496079",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test run variables\n",
    "\n",
    "# Define a verbose flag (set it to True for verbose output)\n",
    "#verbose = True\n",
    "verbose = False\n",
    "\n",
    "source_dir = r'D:\\\\Orthanc'\n",
    "target_dir = r'D:\\\\training' # The output PNG files will be written to the same folder name with _images appended\n",
    "training_ratio, validation_ratio = 0.7, 0.15\n",
    "\n",
    "# Variables to control database and image deletion\n",
    "delete_db = True\n",
    "delete_images = True\n",
    "db_path = 'medical_imaging.db'\n",
    "\n",
    "# Image dimensions\n",
    "img_width, img_height = 152, 152\n",
    "\n",
    "# Training parameters\n",
    "batch_size = 32\n",
    "epochs = 20 # Can increase the epochs since early stopping will handle overfitting\n",
    "threshold = 0.5\n",
    "\n",
    "# File name for saved model\n",
    "model_name = 'lung_ct_classification_model.h5'\n",
    "\n",
    "# Percentage of images to apply message_boxes to\n",
    "message_box_percentage = 100 \n",
    "\n",
    "# Maximum number of images to process (needs to approximately balance the number of valid images)\n",
    "max_invalid_images = 15000 \n",
    "# Maximum number of custom invalid images to process\n",
    "max_custom_invalid_images = 100 \n",
    "# Maximum number of custom valid images to process\n",
    "max_custom_valid_images = 100 \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ada53e62-15fe-4785-904e-46ec7be3f413",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Append '_images' to the end of target_dir\n",
    "training_images_dir = f\"{target_dir}_images\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f95a0008-2899-41ad-a9a3-9499e811cac5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Log to a log file that is specific for the test run and also to the screen if verbose is set\n",
    "\n",
    "class CustomLogger:\n",
    "    def __init__(self, verbose=False):\n",
    "        self.verbose = verbose\n",
    "\n",
    "        # Define log format to include date and time\n",
    "        log_format = '%(asctime)s - %(levelname)s - %(message)s'\n",
    "        log_filename = f'log_{datetime.now().strftime(\"%Y%m%d_%H%M%S\")}.log'\n",
    "\n",
    "        # Configure logging with UTF-8 encoding and specified log format\n",
    "        logging.basicConfig(filename=log_filename, format=log_format, level=logging.INFO, encoding='utf-8', filemode='w', datefmt='%Y-%m-%d %H:%M:%S')\n",
    "\n",
    "    def iprint(self, message):\n",
    "        if self.verbose:\n",
    "            print(message)\n",
    "        logging.info(message)\n",
    "        \n",
    "    def eprint(self, message):\n",
    "        print(message)\n",
    "        logging.error(message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "fc0cb8d8-f886-4d4f-bd1b-29329dc3f97b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def setup_database(db_path):\n",
    "\n",
    "    logger.iprint (\"In function setup_database\")\n",
    "    \n",
    "    # Connect to SQLite database (this will create the database if it doesn't exist)\n",
    "    \n",
    "    try:\n",
    "        conn = sqlite3.connect(db_path)\n",
    "        cursor = conn.cursor()\n",
    "\n",
    "        # Create tables\n",
    "        cursor.execute('''CREATE TABLE IF NOT EXISTS Patients (\n",
    "                            PatientID TEXT PRIMARY KEY,\n",
    "                            PatientInfo TEXT);''')\n",
    "\n",
    "        cursor.execute('''CREATE TABLE IF NOT EXISTS Studies (\n",
    "                            StudyID TEXT PRIMARY KEY,\n",
    "                            PatientID TEXT,\n",
    "                            StudyDate TEXT,\n",
    "                            StudyDescription TEXT,\n",
    "                            BodyPartExamined TEXT,\n",
    "                            FOREIGN KEY (PatientID) REFERENCES Patients (PatientID));''')\n",
    "\n",
    "        cursor.execute('''CREATE TABLE IF NOT EXISTS Series (\n",
    "                            SeriesID TEXT PRIMARY KEY,\n",
    "                            StudyID TEXT,\n",
    "                            SeriesDate TEXT,\n",
    "                            SeriesDescription TEXT,\n",
    "                            Modality TEXT,\n",
    "                            FOREIGN KEY (StudyID) REFERENCES Studies (StudyID));''')\n",
    "        \n",
    "        cursor.execute('''CREATE TABLE IF NOT EXISTS Images (\n",
    "                            ImageID TEXT PRIMARY KEY,\n",
    "                            SeriesID TEXT,\n",
    "                            FilePath TEXT,\n",
    "                            State TEXT,\n",
    "                            InstanceNumber TEXT,\n",
    "                            FOREIGN KEY (SeriesID) REFERENCES Series (SeriesID));''')\n",
    "\n",
    "        cursor.execute('''CREATE TABLE IF NOT EXISTS AugmentedImages (\n",
    "                            AugmentedImageID INTEGER PRIMARY KEY AUTOINCREMENT,\n",
    "                            ImageID TEXT,\n",
    "                            PngFilePath TEXT,\n",
    "                            Transformation TEXT,\n",
    "                            WindowCenter TEXT,\n",
    "                            WindowWidth TEXT,\n",
    "                            RescaleIntercept TEXT,\n",
    "                            RescaleSlope TEXT,\n",
    "                            FOREIGN KEY (ImageID) REFERENCES Images (ImageID));''')\n",
    "        conn.commit()\n",
    "    except sqlite3.DatabaseError as e:\n",
    "        print(f\"Database error: {e}\")\n",
    "    finally:\n",
    "        conn.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d0cc0e11-ec45-49e5-862f-9b9c7c937265",
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_dicom_file(file_path):\n",
    "\n",
    "    logger.iprint (\"In function is_dicom_file\")\n",
    "    \n",
    "    try:\n",
    "        pydicom.dcmread(file_path, stop_before_pixels=True)\n",
    "        return True\n",
    "    except Exception as e:\n",
    "        logger.eprint(f\"Error reading DICOM file {file_path}: {e}\")\n",
    "        return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e9efffed-97a2-49c0-8833-e72816c9b8c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def clear_and_create_directory(directory):\n",
    "\n",
    "    logger.iprint (\"In function clear_and_create_directory\")\n",
    "    \n",
    "    if os.path.exists(directory):\n",
    "        shutil.rmtree(directory)\n",
    "    os.makedirs(directory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4edcccb3-3134-498b-84db-df8545c3690a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def copy_files(file_list, destination):\n",
    "\n",
    "    logger.iprint (\"In function copy_files \")\n",
    "    \n",
    "    try:\n",
    "        os.makedirs(destination, exist_ok=True)\n",
    "        for file_path in file_list:\n",
    "            shutil.copy(file_path, os.path.join(destination, os.path.basename(file_path)))\n",
    "    except (FileNotFoundError, PermissionError) as e:\n",
    "        logger.eprint(f\"File I/O error: {e}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5ef79f8e-be98-4509-96b3-38e6180117ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_metadata(dicom_file_path):\n",
    "    \n",
    "    logger.iprint (\"In function extract_metadata \")\n",
    "    \n",
    "    # Extract metadata like PatientID, StudyID, SeriesID, ImageID, Modality, and BodyPart\n",
    "    ds = pydicom.dcmread(dicom_file_path, stop_before_pixels=True)\n",
    "    patient_id = ds.PatientID\n",
    "    study_id = ds.StudyInstanceUID\n",
    "    series_id = ds.SeriesInstanceUID\n",
    "    image_id = ds.SOPInstanceUID\n",
    "    study_description = ds.StudyDescription if 'StudyDescription' in ds else 'N/A'\n",
    "    series_description = ds.SeriesDescription if 'SeriesDescription' in ds else 'N/A'\n",
    "    instance_number = ds.InstanceNumber if 'InstanceNumber' in ds else 'N/A'\n",
    "    modality = ds.Modality if 'Modality' in ds else 'N/A'\n",
    "    body_part_examined = ds.BodyPartExamined if 'BodyPartExamined' in ds else 'N/A'\n",
    "    study_date = ds.StudyDate if 'StudyDate' in ds else 'N/A'\n",
    "    series_date = ds.SeriesDate if 'SeriesDate' in ds else 'N/A'\n",
    " \n",
    "    return patient_id, study_id, series_id, image_id, modality, body_part_examined, instance_number, study_description, series_description, study_date, series_date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8f1d1262-c3d7-4a69-b134-aa736c9f2d00",
   "metadata": {},
   "outputs": [],
   "source": [
    "def insert_metadata_into_db(cursor, patient_id, study_id, series_id, image_id, modality, body_part_examined, instance_number, study_description, series_description, study_date, series_date, file_path, state):\n",
    "        \n",
    "    logger.iprint (\"In function insert_metadata_into_db\")\n",
    "    \n",
    "    # Insert data into the Patients table\n",
    "    cursor.execute(\"INSERT OR IGNORE INTO Patients (PatientID) VALUES (?)\", (patient_id,))\n",
    "\n",
    "    # Insert data into the Studies table\n",
    "    cursor.execute(\"INSERT OR IGNORE INTO Studies (StudyID, PatientID, StudyDate, StudyDescription, BodyPartExamined) VALUES (?, ?, ?, ?, ?)\",\n",
    "                   (study_id, patient_id, study_date, study_description, body_part_examined))  # Replace \"StudyDate\" with actual values if needed\n",
    "\n",
    "    # Insert data into the Series table\n",
    "    cursor.execute(\"INSERT OR IGNORE INTO Series (SeriesID, StudyID, SeriesDate, SeriesDescription, Modality) VALUES (?, ?, ?, ?, ?)\",\n",
    "                   (series_id, study_id, series_date, series_description, modality))  # Replace \"SeriesDate\" with actual values if needed\n",
    "\n",
    "    # Insert data into the Images table\n",
    "    cursor.execute(\"INSERT OR IGNORE INTO Images (ImageID, SeriesID, FilePath, State, InstanceNumber) VALUES (?, ?, ?, ?, ?)\", \n",
    "                   (image_id, series_id, file_path, state, instance_number))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1c1ee840-7d8c-48f2-a5d5-4214c3059a2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_rescale_and_window_level(dcm):\n",
    "    \"\"\"\n",
    "    Apply the rescale slope and intercept, and window center and width to the DICOM image data.\n",
    "\n",
    "    Parameters:\n",
    "    - dcm: DICOM dataset.\n",
    "\n",
    "    Returns:\n",
    "    - rescaled_and_windowed_image: The image after applying rescale and window level.\n",
    "    - window_center: Window center used for windowing.\n",
    "    - window_width: Window width used for windowing.\n",
    "    - rescale_slope: Rescale slope used for rescaling.\n",
    "    - rescale_intercept: Rescale intercept used for rescaling.\n",
    "    \"\"\"\n",
    "    \n",
    "    logger.iprint (\"In function apply_rescale_and_window_level\")    \n",
    "    \n",
    "    # Apply rescale slope and intercept if available\n",
    "    rescale_slope = getattr(dcm, 'RescaleSlope', 1)  # Default to 1 if not present\n",
    "    rescale_intercept = getattr(dcm, 'RescaleIntercept', 0)  # Default to 0 if not present\n",
    "    rescaled_image = dcm.pixel_array.astype(np.float64) * rescale_slope + rescale_intercept\n",
    "\n",
    "    # Apply window center and width if available\n",
    "    if hasattr(dcm, 'WindowCenter') and hasattr(dcm, 'WindowWidth'):\n",
    "        window_center = dcm.WindowCenter\n",
    "        window_width = dcm.WindowWidth\n",
    "        if isinstance(window_center, pydicom.multival.MultiValue):\n",
    "            window_center = window_center[0]\n",
    "        if isinstance(window_width, pydicom.multival.MultiValue):\n",
    "            window_width = window_width[0]\n",
    "        window_center = float(window_center)\n",
    "        window_width = float(window_width)\n",
    "\n",
    "        low = window_center - window_width / 2\n",
    "        high = window_center + window_width / 2\n",
    "        rescaled_and_windowed_image = np.clip(rescaled_image, low, high)\n",
    "    else:\n",
    "        # If window level is not specified, use the rescaled image\n",
    "        rescaled_and_windowed_image = rescaled_image\n",
    "        window_center = None\n",
    "        window_width = None\n",
    "\n",
    "    return rescaled_and_windowed_image, window_center, window_width, rescale_slope, rescale_intercept"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6ddfef1e-93e7-40c6-a24f-7abb6ddca8f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalize_image(image):\n",
    "    \n",
    "    logger.iprint (\"In function normalize_image\")    \n",
    "    \n",
    "    \"\"\"Normalize the image data to 0-255 and convert to uint8.\"\"\"\n",
    "    image = image - np.min(image)\n",
    "    image = image / np.max(image)\n",
    "    image = (image * 255).astype(np.uint8)\n",
    "    return image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ddfc47b4-7fa7-497d-b82a-aa6549a5c986",
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_rotate_and_flip(image):\n",
    "  \n",
    "    logger.iprint (\"In function random_rotate_and_flip\")    \n",
    "  \n",
    "    try:\n",
    "        # Convert PIL Image to numpy array for processing\n",
    "        image_np = np.array(image)\n",
    "\n",
    "        # Random rotation by 90, 180, or 270 degrees\n",
    "        rotations = [0, 90, 180, 270]\n",
    "        rotation_choice = random.choice(rotations)\n",
    "        if rotation_choice != 0:\n",
    "            image_np = np.rot90(image_np, rotation_choice // 90)  # np.rot90 expects k=1,2,3 for 90,180,270 degrees\n",
    "\n",
    "        # Random flip\n",
    "        if random.choice([True, False]):\n",
    "            image_np = np.fliplr(image_np)\n",
    "        if random.choice([True, False]):\n",
    "            image_np = np.flipud(image_np)\n",
    "\n",
    "        # Convert numpy array back to PIL Image\n",
    "        rotated_and_flipped_image = Image.fromarray(image_np)\n",
    "\n",
    "        return rotated_and_flipped_image\n",
    "    except Exception as e:\n",
    "        # Log the error or print it out\n",
    "        logger.eprint(\"Error during rotation and flip: \" + e)\n",
    "        # Return the original image in case of error\n",
    "        return image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f0c7e0b9-63ec-4a44-9374-22dd76be80c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_zoom(image):\n",
    "     \n",
    "    logger.iprint (\"In function random_zoom\")    \n",
    " \n",
    "    # Randomly choose a zoom factor from 80% to 120%\n",
    "    zoom_factor = random.uniform(0.8, 1.2)\n",
    "    width, height = image.size\n",
    "    \n",
    "    # Calculate the new width and height based on the zoom factor\n",
    "    new_width = int(width * zoom_factor)\n",
    "    new_height = int(height * zoom_factor)\n",
    "    \n",
    "    # Resize the image\n",
    "    resized_image = image.resize((new_width, new_height), Image.Resampling.LANCZOS)\n",
    "    \n",
    "    # If zooming out, pad the image with black pixels\n",
    "    if zoom_factor < 1.0:\n",
    "        delta_w = width - new_width\n",
    "        delta_h = height - new_height\n",
    "        padding = (delta_w // 2, delta_h // 2, delta_w - (delta_w // 2), delta_h - (delta_h // 2))\n",
    "        return ImageOps.expand(resized_image, padding, fill=0)\n",
    "    elif zoom_factor > 1.0:\n",
    "        # If zooming in, crop the image to the original size\n",
    "        return resized_image.crop(((new_width - width) // 2,\n",
    "                                   (new_height - height) // 2,\n",
    "                                   (new_width + width) // 2,\n",
    "                                   (new_height + height) // 2))\n",
    "    else:\n",
    "        # If zoom_factor is 1, return the original image\n",
    "        return image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "5236179c-f45f-4c05-8661-eb7ec2a69ceb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_blur(image):\n",
    "     \n",
    "    logger.iprint (\"In function random_blur\")    \n",
    " \n",
    "    # Randomly choose a blur radius from 0 to 5\n",
    "    blur_radius = random.uniform(0, 5)\n",
    "    \n",
    "    # Apply Gaussian blur with the chosen radius\n",
    "    blurred_image = image.filter(ImageFilter.GaussianBlur(blur_radius))\n",
    "    \n",
    "    return blurred_image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0093f74a-cff0-470b-8716-ad3e267dc6cd",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9f0c3c06-8c92-44fb-badf-2f5c0f575a3c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def construct_transformation_description(randomized_wl, blurred, rotate_and_flip, zoom, invert):\n",
    "  \n",
    "    logger.iprint (\"In function construct_transformation_description\")    \n",
    "    \n",
    "    transformations = []\n",
    "\n",
    "    if randomized_wl:\n",
    "        transformations.append(\"randomized_window_level\")\n",
    "    if blurred:\n",
    "        transformations.append(\"blurred\")\n",
    "    if rotate_and_flip:\n",
    "        transformations.append(\"rotated_flipped\")\n",
    "    if zoom:\n",
    "        transformations.append(\"zoomed\")\n",
    "    if invert:\n",
    "        transformations.append(\"invert\")\n",
    "\n",
    "    # Join all transformations with a plus sign or another separator that makes sense in your context\n",
    "    transformation_description = \"+\".join(transformations) or \"original\"\n",
    "\n",
    "    return transformation_description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f1378529-d69f-44f3-a51c-a32443c230c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_window(image, min_level=-700, max_level=100, min_width=1200, max_width=2000):\n",
    "\n",
    "    logger.iprint (\"In function random_window\")    \n",
    "    \n",
    "    window_level = random.randint(min_level, max_level)\n",
    "    window_width = random.randint(min_width, max_width)\n",
    "\n",
    "    img_min = window_level - window_width / 2\n",
    "    img_max = window_level + window_width / 2\n",
    "    windowed_img = np.clip(image, img_min, img_max)\n",
    "\n",
    "    windowed_img -= windowed_img.min()\n",
    "    windowed_img /= windowed_img.max()\n",
    "    return windowed_img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "f7ff2a9f-9539-4d6e-8174-01ed13298bc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import ImageOps\n",
    "\n",
    "def convert_dicom_to_png(dicom_dir, output_base_dir, state, cursor, randomized_wl=False, blurred=False, rotate_and_flip=False, zoom=False, invert=False):\n",
    "    logger.iprint(\"In function convert_dicom_to_png\")\n",
    "\n",
    "    try:\n",
    "        output_dir = os.path.join(output_base_dir, \n",
    "                                  \"zoomed\" if zoom else \"\",\n",
    "                                  \"blurred\" if blurred else \"\", \n",
    "                                  \"randomized_wl\" if randomized_wl else \"\",\n",
    "                                  \"rotate_and_flip\" if rotate_and_flip else \"\",\n",
    "                                  \"inverted\" if invert else \"\")\n",
    "        os.makedirs(output_dir, exist_ok=True)\n",
    "        \n",
    "        transformation_description = construct_transformation_description(randomized_wl, blurred, rotate_and_flip, zoom, invert)\n",
    "  \n",
    "        for entry in os.listdir(dicom_dir):\n",
    "            logger.iprint(\"Processing DICOM file: \" + entry)\n",
    "            dicom_path = os.path.join(dicom_dir, entry)\n",
    "\n",
    "            if os.path.isfile(dicom_path):\n",
    "                try:\n",
    "                    dcm = pydicom.dcmread(dicom_path)\n",
    "                    if not hasattr(dcm, 'PixelData') and not hasattr(dcm, 'FloatPixelData') and not hasattr(dcm, 'DoubleFloatPixelData'):\n",
    "                        raise ValueError(\"DICOM file does not contain image pixel data\")\n",
    "                    \n",
    "                    rescaled_and_windowed_image, window_center, window_width, rescale_slope, rescale_intercept = apply_rescale_and_window_level(dcm)\n",
    "\n",
    "                    normalized_image = normalize_image(rescaled_and_windowed_image)\n",
    "\n",
    "                    pil_image = Image.fromarray(normalized_image.astype(np.uint8))\n",
    "\n",
    "                    if zoom:\n",
    "                        pil_image = random_zoom(pil_image)\n",
    "                    if blurred:\n",
    "                        pil_image = random_blur(pil_image)\n",
    "                    if rotate_and_flip:\n",
    "                        pil_image = random_rotate_and_flip(pil_image)\n",
    "                    if invert:\n",
    "                        pil_image = ImageOps.invert(pil_image)\n",
    "\n",
    "                    png_filename = entry + '.png'\n",
    "                    png_path = os.path.join(output_dir, png_filename)\n",
    "                    pil_image.save(png_path)\n",
    "                    \n",
    "                    cursor.execute(\"INSERT INTO AugmentedImages (ImageID, PngFilePath, Transformation, WindowCenter, WindowWidth, RescaleIntercept, RescaleSlope) VALUES (?, ?, ?, ?, ?, ?, ?)\", \n",
    "                                   (dcm.SOPInstanceUID, png_path, transformation_description, window_center, window_width, rescale_intercept, rescale_slope))\n",
    "                    \n",
    "                except Exception as e:\n",
    "                    logger.iprint(f\"Failed to convert: {dicom_path}, Error: {e}\")\n",
    "\n",
    "    except Exception as e:\n",
    "        logger.eprint(f\"Unexpected error: {e}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "65f3a3f8-b4ff-40a7-bc81-6d1dc72af27b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_error_message():\n",
    "\n",
    "    logger.iprint (\"In function generate_error_message\")    \n",
    "\n",
    "    base_messages = [\n",
    "        \"Lorem ipsum dolor sit amet, consectetur adipiscing elit.\",\n",
    "        \"Error: The operation could not be completed.\",\n",
    "        \"Warning: System memory is running low.\",\n",
    "        \"Alert: Unrecognized device detected.\",        \n",
    "        \"System failure: Please contact support.\",\n",
    "        \"Update failed: Please retry or check your connection.\"\n",
    "    ]\n",
    "    # Select a base message randomly\n",
    "    message = random.choice(base_messages)\n",
    "    \n",
    "    # Optionally append a random error code or numerical detail\n",
    "    if random.choice([True, False]):  # Decide randomly whether to add a number\n",
    "        number = random.randint(100, 9999)\n",
    "        # Randomly choose how to add the number (end as a code or within the text)\n",
    "        if random.choice([True, False]):\n",
    "            message += f\" Code: {number}.\"\n",
    "        else:\n",
    "            parts = message.split()\n",
    "            insert_at = random.randint(1, len(parts) - 1)\n",
    "            parts.insert(insert_at, str(number))\n",
    "            message = ' '.join(parts)\n",
    "    \n",
    "    return message"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "581e5c1b-d04d-4dea-aa86-f8b5f1a207cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_error_message_box(image_path, output_path, font_paths, message_box_percentage):\n",
    "\n",
    "    logger.iprint (\"In function apply_error_message_box\")    \n",
    "    \n",
    "    if random.randint(1, 100) > message_box_percentage:\n",
    "        # Skip this image; do not apply message_box\n",
    "        return\n",
    "\n",
    "    image = Image.open(image_path).convert('L')  # Ensure image is in grayscale\n",
    "    draw = ImageDraw.Draw(image)\n",
    "\n",
    "    # Random font and size\n",
    "    font_path = random.choice(font_paths)\n",
    "    font_size = random.randint(15, 25)  # Adjust range as needed\n",
    "    font = ImageFont.truetype(font_path, font_size)\n",
    "\n",
    "    # Generate message\n",
    "    message = generate_error_message()\n",
    "\n",
    "    # Wrap the message\n",
    "    wrapped_text = \"\\n\".join(textwrap.wrap(message, width=40))\n",
    "\n",
    "    # Calculate the bounding box for the wrapped text\n",
    "    text_bbox = draw.textbbox((0, 0), wrapped_text, font=font)\n",
    "    text_width = text_bbox[2] - text_bbox[0]\n",
    "    text_height = text_bbox[3] - text_bbox[1]\n",
    "\n",
    "    # Determine box size based on text dimensions, making box_height randomly larger\n",
    "    box_width = max(text_width + 20, image.width // 4)  # Ensure minimum width and add padding\n",
    "    # Randomize the height a bit more significantly\n",
    "    box_height = random.randint(text_height + 20, int(1.5 * image.height // 4))\n",
    "\n",
    "    # Randomly calculate the box position to fit within the image\n",
    "    max_x = max(image.width - box_width, 0)\n",
    "    max_y = max(image.height - box_height, 0)\n",
    "    box_x = random.randint(0, max_x)\n",
    "    box_y = random.randint(0, max_y)\n",
    "\n",
    "    # Choose a random grayscale value for the box fill\n",
    "    box_fill = random.randint(0, 255)\n",
    "    # Use white text if the box is dark, black otherwise\n",
    "    text_color = 255 if box_fill < 128 else 0\n",
    "\n",
    "    # Draw the message box\n",
    "    draw.rectangle([box_x, box_y, box_x + box_width, box_y + box_height], fill=box_fill)\n",
    "\n",
    "    # Draw the text, left and top justified within the box\n",
    "    # By starting the text at the top left corner of the box (with a little padding)\n",
    "    text_start_x = box_x + 10  # Add some padding from the left edge\n",
    "    text_start_y = box_y + 10  # Add some padding from the top edge\n",
    "    draw.text((text_start_x, text_start_y), wrapped_text, fill=text_color, font=font, align='left')\n",
    "\n",
    "    # Save the modified image\n",
    "    image.save(output_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "2b1e3424-d3b5-496c-affa-8176fefcf68a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def augment_images_with_message_boxes(source_dir, output_dir, message_box_percentage, font_path=\"arial.ttf\"):\n",
    "\n",
    "    logger.iprint (\"In function augment_images_with_message_boxes\")    \n",
    "    \n",
    "    if not os.path.exists(output_dir):\n",
    "        os.makedirs(output_dir)\n",
    "    \n",
    "    all_images = os.listdir(source_dir)\n",
    "    num_images_to_augment = int(len(all_images) * (message_box_percentage / 100.0))\n",
    "    images_to_augment = random.sample(all_images, num_images_to_augment)\n",
    "    logger.iprint (\"Num images to add message boxes to: \" + num_images_to_augment)\n",
    "\n",
    "    for image_file in images_to_augment:\n",
    "        logger.iprint (image_file)\n",
    "        if image_file.endswith(('.png', '.jpg', '.jpeg')):\n",
    "            image_path = os.path.join(source_dir, image_file)\n",
    "            output_path = os.path.join(output_dir, \"message_box_\" + image_file)\n",
    "            apply_error_message_box(image_path, output_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "e1575453-a8bc-43ca-849f-d2d6d95a4afb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def copy_random_images(source_dir, target_dir, class_name, max_images=None, train_ratio=0.7, validate_ratio=0.15, test_ratio=0.15, subfolder_name=None):\n",
    "\n",
    "    logger.iprint (\"In function copy_random_images\")    \n",
    "  \n",
    "    \"\"\"\n",
    "    Copies a random subset of images from the source directory to the train, validate, and test directories within the target directory.\n",
    "    Each set will be placed in a subdirectory named after the specified class, and optionally, within a further specified subfolder.\n",
    "\n",
    "    Parameters:\n",
    "    - source_dir: The directory containing the original images.\n",
    "    - target_dir: The base directory where the train, validate, and test directories will be created.\n",
    "    - class_name: The class name (e.g., 'valid' or 'invalid') specifying the subdirectory within train, validate, and test where images should be copied.\n",
    "    - max_images: The maximum number of images to copy. If None, all images in source_dir are considered.\n",
    "    - train_ratio: The fraction of images to copy to the training set.\n",
    "    - validate_ratio: The fraction of images to copy to the validation set.\n",
    "    - test_ratio: The fraction of images to copy to the testing set.\n",
    "    - subfolder_name: Optional name of a subfolder within each set directory where images will be copied.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Ensure the ratios sum to 1\n",
    "    assert train_ratio + validate_ratio + test_ratio == 1, \"Ratios must sum to 1\"\n",
    "    \n",
    "    # Recursively get all image filenames in the source directory and its subdirectories\n",
    "    all_images = []\n",
    "    for root, dirs, files in os.walk(source_dir):\n",
    "        for file in files:\n",
    "            \n",
    "            if os.path.isfile(os.path.join(root, file)):\n",
    "                all_images.append(os.path.join(root, file))\n",
    "    \n",
    "    if max_images is not None and max_images < len(all_images):\n",
    "        all_images = sample(all_images, max_images)\n",
    "    \n",
    "    # Calculate the number of images for each set\n",
    "    num_train = int(len(all_images) * train_ratio)\n",
    "    num_validate = int(len(all_images) * validate_ratio)\n",
    "    num_test = len(all_images) - num_train - num_validate\n",
    "    \n",
    "    # Randomly select images for each set\n",
    "    train_images = sample(all_images, num_train)\n",
    "    validate_images = sample([img for img in all_images if img not in train_images], num_validate)\n",
    "    test_images = [img for img in all_images if img not in train_images + validate_images]\n",
    "    \n",
    "    def copy_images(images, target_subdir):\n",
    "        if subfolder_name:\n",
    "            target_subdir = os.path.join(target_subdir, subfolder_name)\n",
    "        os.makedirs(target_subdir, exist_ok=True)\n",
    "        for img_path in images:\n",
    "            filename = os.path.basename(img_path)\n",
    "            target_file_path = os.path.join(target_subdir, filename)\n",
    "            if os.path.abspath(img_path) != os.path.abspath(target_file_path):\n",
    "                shutil.copy(img_path, target_file_path)\n",
    "    \n",
    "    # Copy images to their respective directories\n",
    "    copy_images(train_images, os.path.join(target_dir, f'train\\\\{class_name}'))\n",
    "    copy_images(validate_images, os.path.join(target_dir, f'validate\\\\{class_name}'))\n",
    "    copy_images(test_images, os.path.join(target_dir, f'test\\\\{class_name}'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "d3135699-a912-4ba2-88dd-c525dc727fe3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_message_boxes_to_dataset(source_base_dir, output_base_dir, subset, font_paths, message_box_percentage):\n",
    "\n",
    "    logger.iprint (\"In function apply_message_boxes_to_dataset\")    \n",
    "    \n",
    "    source_dir = os.path.join(source_base_dir, subset, 'valid')\n",
    "    output_dir = os.path.join(output_base_dir, subset, 'invalid', 'message_boxes')\n",
    "    \n",
    "    if not os.path.exists(output_dir):\n",
    "        os.makedirs(output_dir)\n",
    "    \n",
    "    for image_file in os.listdir(source_dir):\n",
    "        if image_file.endswith(('.png', '.jpg', '.jpeg')):\n",
    "            image_path = os.path.join(source_dir, image_file)\n",
    "            output_path = os.path.join(output_dir, \"message_box_\" + image_file)\n",
    "            apply_error_message_box(image_path, output_path, font_paths, message_box_percentage)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "02ed1141-b600-4963-8823-0c3895912603",
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_dicom_file(file_path):\n",
    "    \n",
    "    logger.iprint (\"In function is_dicom_file\")    \n",
    "    \n",
    "    try:\n",
    "        pydicom.dcmread(file_path, stop_before_pixels=True)\n",
    "        return True\n",
    "    except:\n",
    "        return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "994f4c24-ca6d-47b9-bf6e-6520ff639a4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to display the false negative image\n",
    "def display_fn_image(image_path):\n",
    " \n",
    "    logger.iprint (\"In function display_fn_image\") \n",
    "    \n",
    "    image = load_img(image_path, color_mode='grayscale')\n",
    "    plt.figure(figsize=(4, 4))\n",
    "    plt.imshow(image, cmap='gray')\n",
    "    plt.title(\"False Negative\")\n",
    "    plt.axis('off')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "998758ad-c040-4db1-b714-d905fe1e2f0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to display the false positive image\n",
    "def display_fp_image(image_path):\n",
    "     \n",
    "    logger.iprint (\"In function display_fp_image\") \n",
    "    \n",
    "    image = plt.imread(image_path)\n",
    "    plt.figure(figsize=(5, 5))\n",
    "    plt.imshow(image, cmap='gray')\n",
    "    plt.title(\"False Positive\")\n",
    "    plt.axis('off')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "154cc06c-cb4c-456a-8fde-1e21b135312f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_image(image_path, img_width, img_height):\n",
    "\n",
    "    logger.iprint (\"In function preprocess_image\") \n",
    "   \n",
    "    image = load_img(image_path, target_size=(img_width, img_height), color_mode='grayscale')\n",
    "    image = img_to_array(image)\n",
    "    image = np.expand_dims(image, axis=0)  # Add batch dimension\n",
    "    image /= 255.0  # Normalize to [0, 1]\n",
    "    return image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "fb0c62a2-3362-4772-816b-2dbde893ef5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Main code execution for image pipeline\n",
    "\n",
    "logger = CustomLogger(verbose)\n",
    "\n",
    "# Check if the database exists and delete it if delete_db is True\n",
    "if delete_db and os.path.exists(db_path):\n",
    "    os.remove(db_path)\n",
    "    logger.iprint(\"Existing database removed.\")\n",
    "else:\n",
    "    logger.iprint(\"Skipping database removal.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "0adfb0ab-86f6-4997-891d-3e1d8f0e0306",
   "metadata": {},
   "outputs": [],
   "source": [
    "setup_database(db_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "e89a1ace-23f8-44ff-83e2-f048d1b57586",
   "metadata": {},
   "outputs": [],
   "source": [
    "setup_database(db_path)\n",
    "# In your main function or processing script\n",
    "conn = sqlite3.connect(db_path)\n",
    "cursor = conn.cursor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "df1e0993-fe0f-4a6a-ac9a-592cde703320",
   "metadata": {},
   "outputs": [],
   "source": [
    "if delete_images and (os.path.exists(target_dir) or os.path.exists(training_images_dir)):\n",
    "    clear_and_create_directory(target_dir)\n",
    "    clear_and_create_directory(training_images_dir)\n",
    "    logger.iprint(\"Existing images removed.\")\n",
    "else:\n",
    "    logger.iprint(\"Skipping image removal.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "e28f8e68-85a0-475d-be97-bcf375d3cf11",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy the DIDOM files from the image archive and add them to the database\n",
    "dicom_files = []\n",
    "for root, dirs, files in os.walk(source_dir):\n",
    "    for file in files:\n",
    "        file_path = os.path.join(root, file)\n",
    "        if is_dicom_file(file_path):\n",
    "            dicom_files.append(file_path)\n",
    "\n",
    "random.shuffle(dicom_files)\n",
    "total_files = len(dicom_files)\n",
    "training_count = int(total_files * training_ratio)\n",
    "validation_count = int(total_files * validation_ratio)\n",
    "\n",
    "training_files = dicom_files[:training_count]\n",
    "validation_files = dicom_files[training_count:training_count + validation_count]\n",
    "test_files = dicom_files[training_count + validation_count:]\n",
    "\n",
    "for file_path in training_files:\n",
    "    patient_id, study_id, series_id, image_id, modality, body_part_examined, instance_number, study_description, series_description, study_date, series_date = extract_metadata(file_path)\n",
    "    insert_metadata_into_db(cursor, patient_id, study_id, series_id, image_id, modality, body_part_examined, instance_number, study_description, series_description, study_date, series_date, file_path, 'train')\n",
    "\n",
    "for file_path in validation_files:\n",
    "    patient_id, study_id, series_id, image_id, modality, body_part_examined, instance_number, study_description, series_description, study_date, series_date = extract_metadata(file_path)\n",
    "    insert_metadata_into_db(cursor, patient_id, study_id, series_id, image_id, modality, body_part_examined, instance_number, study_description, series_description, study_date, series_date, file_path, 'validate')\n",
    "\n",
    "for file_path in test_files:\n",
    "    patient_id, study_id, series_id, image_id, modality, body_part_examined, instance_number, study_description, series_description, study_date, series_date = extract_metadata(file_path)\n",
    "    insert_metadata_into_db(cursor, patient_id, study_id, series_id, image_id, modality, body_part_examined, instance_number, study_description, series_description, study_date, series_date, file_path, 'test')\n",
    "\n",
    "conn.commit()\n",
    "\n",
    "copy_files(training_files, os.path.join(target_dir, 'train\\\\valid'))\n",
    "copy_files(validation_files, os.path.join(target_dir, 'validate\\\\valid'))\n",
    "copy_files(test_files, os.path.join(target_dir, 'test\\\\valid'))\n",
    "\n",
    "logger.iprint(f\"Total DICOM files: \" + str(total_files))\n",
    "logger.iprint(f\"Training files: \" + str(len(training_files)))\n",
    "logger.iprint(f\"Validation files: \" + str(len(validation_files)))\n",
    "logger.iprint(f\"Test files: \" + str(len(test_files)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f824ef3-b7e4-4fbd-86fc-236ac1393205",
   "metadata": {},
   "outputs": [],
   "source": [
    "# After inserting DICOM metadata...\n",
    "# Create the PNG files for training based on the DICOM tags\n",
    "randomized_wl=False\n",
    "blurred=False\n",
    "rotate_and_flip=False\n",
    "zoomed=False\n",
    "invert=False\n",
    "\n",
    "convert_dicom_to_png(os.path.join(target_dir, 'train\\\\valid'), os.path.join(target_dir, '..', 'training_images', 'train', 'valid'), 'train', cursor, randomized_wl, blurred, rotate_and_flip, zoomed, invert)\n",
    "convert_dicom_to_png(os.path.join(target_dir, 'validate\\\\valid'), os.path.join(target_dir, '..', 'training_images', 'validate', 'valid'), 'validate', cursor, randomized_wl, blurred, rotate_and_flip, zoomed, invert)\n",
    "convert_dicom_to_png(os.path.join(target_dir, 'test\\\\valid'),os.path.join(target_dir, '..', 'training_images', 'test', 'valid'), 'test', cursor, randomized_wl, blurred, rotate_and_flip, zoomed, invert)\n",
    "\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4977f639-d0af-4bef-b288-6420c9f0ce1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The PNGs have been created from the original DICOM images \n",
    "# Now create copies of the same files with randomized window levels to reduce overfitting\n",
    "randomized_wl=True\n",
    "blurred=False\n",
    "rotate_and_flip=False\n",
    "zoomed=False\n",
    "invert=False\n",
    "\n",
    "convert_dicom_to_png(os.path.join(target_dir, 'train\\\\valid'), os.path.join(target_dir, '..', 'training_images', 'train', 'valid'), 'train', cursor, randomized_wl, blurred, rotate_and_flip, zoomed, invert)\n",
    "convert_dicom_to_png(os.path.join(target_dir, 'validate\\\\valid'), os.path.join(target_dir, '..', 'training_images', 'validate', 'valid'), 'validate', cursor, randomized_wl, blurred, rotate_and_flip, zoomed, invert)\n",
    "convert_dicom_to_png(os.path.join(target_dir, 'test\\\\valid'),os.path.join(target_dir, '..', 'training_images', 'test', 'valid'), 'test', cursor, randomized_wl, blurred, rotate_and_flip, zoomed, invert)\n",
    "\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09384af1-b2c1-44e4-9106-a652793a5577",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now create copies of the same files with randomized window levels to reduce overfitting\n",
    "from scipy.ndimage import gaussian_filter\n",
    "\n",
    "randomized_wl=False\n",
    "blurred=True\n",
    "rotate_and_flip=False\n",
    "zoomed=False\n",
    "invert=False\n",
    "\n",
    "convert_dicom_to_png(os.path.join(target_dir, 'train\\\\valid'), os.path.join(target_dir, '..', 'training_images', 'train', 'valid'), 'train', cursor, randomized_wl, blurred, rotate_and_flip, zoomed, invert)\n",
    "convert_dicom_to_png(os.path.join(target_dir, 'validate\\\\valid'), os.path.join(target_dir, '..', 'training_images', 'validate', 'valid'), 'validate', cursor, randomized_wl, blurred, rotate_and_flip, zoomed, invert)\n",
    "convert_dicom_to_png(os.path.join(target_dir, 'test\\\\valid'),os.path.join(target_dir, '..', 'training_images', 'test', 'valid'), 'test', cursor, randomized_wl, blurred, rotate_and_flip, zoomed, invert)\n",
    "\n",
    "conn.commit()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9c3cae7-7adf-4ada-934b-bd3980a8d946",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Now create copies of the same files with randomized rotation and flipping\n",
    "\n",
    "randomized_wl=False\n",
    "blurred=False\n",
    "rotate_and_flip=True\n",
    "zoomed=False\n",
    "invert=False\n",
    "\n",
    "convert_dicom_to_png(os.path.join(target_dir, 'train\\\\valid'), os.path.join(target_dir, '..', 'training_images', 'train', 'valid'), 'train', cursor, randomized_wl, blurred, rotate_and_flip, zoomed, invert)\n",
    "convert_dicom_to_png(os.path.join(target_dir, 'validate\\\\valid'), os.path.join(target_dir, '..', 'training_images', 'validate', 'valid'), 'validate', cursor, randomized_wl, blurred, rotate_and_flip, zoomed, invert)\n",
    "convert_dicom_to_png(os.path.join(target_dir, 'test\\\\valid'),os.path.join(target_dir, '..', 'training_images', 'test', 'valid'), 'test', cursor, randomized_wl, blurred, rotate_and_flip, zoomed, invert)\n",
    "\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d54f6773-160c-479d-82df-4a8e7f39e427",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now create copies of the same files with randomized zoom as well as window level to simulate some internet studies\n",
    "from scipy.ndimage import gaussian_filter\n",
    "\n",
    "randomized_wl=True\n",
    "blurred=False\n",
    "rotate_and_flip=False\n",
    "zoomed=True\n",
    "invert=False\n",
    "\n",
    "convert_dicom_to_png(os.path.join(target_dir, 'train\\\\valid'), os.path.join(target_dir, '..', 'training_images', 'train', 'valid'), 'train', cursor, randomized_wl, blurred, rotate_and_flip, zoomed, invert)\n",
    "convert_dicom_to_png(os.path.join(target_dir, 'validate\\\\valid'), os.path.join(target_dir, '..', 'training_images', 'validate', 'valid'), 'validate', cursor, randomized_wl, blurred, rotate_and_flip, zoomed, invert)\n",
    "convert_dicom_to_png(os.path.join(target_dir, 'test\\\\valid'),os.path.join(target_dir, '..', 'training_images', 'test', 'valid'), 'test', cursor, randomized_wl, blurred, rotate_and_flip, zoomed, invert)\n",
    "\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "405cc7cc-2fad-4fa5-a096-4b2d3198eb7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Database updates complete\n",
    "conn.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f968d39-c788-48c7-95e9-2f7dffa8d47f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy some random invalid Kaggle real and fake images\n",
    "source_dir = 'D:\\\\Kaggle_real_and_fake_images'\n",
    "copy_random_images(source_dir, training_images_dir, 'invalid', max_invalid_images)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee8bf02b-b588-4804-94d4-25760595d86d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy some random custom invalid images\n",
    "source_dir = 'D:\\\\Custom_invalid'\n",
    "copy_random_images(source_dir, training_images_dir, 'invalid', max_custom_invalid_images)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b790496-ff3f-4e40-b24e-b2fbe6aa13a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy some random custom valid images (from internet sources) for testing only\n",
    "source_dir = 'D:\\\\Custom_test_valid'\n",
    "copy_random_images(source_dir, training_images_dir, 'valid', max_custom_valid_images, 0, 0, 1, \"InternetTest\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5796e630-bc0d-44fe-b1af-15a324ed0811",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add message_boxes to images\n",
    "# Note: In this case we are reusing images already selected for training, validation and test\n",
    "# a better approach might be to use images that were not selected\n",
    "\n",
    "font_paths = ['arial.ttf', 'times.ttf']  # Add paths to any other fonts you'd like to use\n",
    "\n",
    "# Apply message_boxes to each dataset subset\n",
    "for subset in ['train', 'validate', 'test']:\n",
    "    apply_message_boxes_to_dataset(training_images_dir, training_images_dir, subset, font_paths, message_box_percentage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bd8c6e4-cf38-48fc-9629-620510cc8a0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# increase the number of message box images as with 100% get some failures\n",
    "\n",
    "# Apply message_boxes to each dataset subset\n",
    "for subset in ['train', 'validate', 'test']:\n",
    "    apply_message_boxes_to_dataset(training_images_dir, training_images_dir, subset, font_paths, message_box_percentage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84d28850-be7b-440b-8afe-bf5d6080dfa7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The following code creates and tests the model based on the image pipeline \n",
    "\n",
    "# Rescale images from [0, 255] to [0, 1]\n",
    "datagen = ImageDataGenerator(rescale=1. / 255)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c39a7d23-bf87-4abf-b5db-e5bd9486a3c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data_dir = f'{target_dir}_images\\\\train'\n",
    "\n",
    "train_generator = datagen.flow_from_directory(\n",
    "    directory=train_data_dir,\n",
    "    target_size=(img_width, img_height),  # Ensure this matches your image dimensions\n",
    "    batch_size=batch_size,\n",
    "    class_mode='binary',  # Assuming a binary classification setup\n",
    "    color_mode='grayscale',  # Assuming grayscale images\n",
    "    shuffle=True\n",
    ")\n",
    "\n",
    "# This will print out the image counts, confirming that images from 'valid' and 'invalid' are included\n",
    "logger.iprint(\"Training found \" + str(train_generator.samples) + \" images belonging to \" + str(train_generator.num_classes) + \" classes.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04163d5a-82a8-46ca-a711-b7fd9fde7801",
   "metadata": {},
   "outputs": [],
   "source": [
    "val_data_dir = f'{target_dir}_images\\\\validate'\n",
    "\n",
    "val_generator = datagen.flow_from_directory(\n",
    "    directory=val_data_dir,\n",
    "    target_size=(img_width, img_height),  # Ensure this matches your image dimensions\n",
    "    batch_size=batch_size,\n",
    "    class_mode='binary',  # Assuming a binary classification setup\n",
    "    color_mode='grayscale',  # Assuming grayscale images\n",
    "    shuffle=True\n",
    ")\n",
    "\n",
    "# This will print out the counts, confirming that images from 'valid' and 'invalid' are included\n",
    "logger.iprint(\"Validation found \" + str(val_generator.samples) + \" images belonging to \" + str(val_generator.num_classes) + \" classes.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "760d7f5e-b187-4b3b-859a-556381b2ad0b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00de1e0b-f5e3-47fe-9ef7-50e7d105bf49",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the model architecture\n",
    "input_img = Input(shape=(img_width, img_height, 1))  # 1 channel for grayscale images\n",
    "x = Conv2D(32, (3, 3), activation='relu', padding='same')(input_img)\n",
    "x = MaxPooling2D((2, 2), padding='same')(x)\n",
    "x = Dropout(0.25)(x)  # Dropout layer after MaxPooling\n",
    "x = Conv2D(64, (3, 3), activation='relu', padding='same')(x)\n",
    "x = MaxPooling2D((2, 2), padding='same')(x)\n",
    "x = Dropout(0.25)(x)  # Dropout layer after MaxPooling\n",
    "x = Flatten()(x)\n",
    "x = Dense(64, activation='relu')(x)\n",
    "x = Dropout(0.5)(x)  # Dropout layer before the output layer\n",
    "classifier_output = Dense(1, activation='sigmoid', name='classifier_output')(x)  # Use sigmoid for binary classification\n",
    "\n",
    "# Create and compile the model\n",
    "model = Model(inputs=input_img, outputs=classifier_output)\n",
    "model.compile(optimizer='adam',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "244a5670-1c3d-4ec9-af7e-21deb64ac85c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# EarlyStopping callback\n",
    "early_stopping = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)\n",
    "\n",
    "# Train the model with EarlyStopping\n",
    "model.fit(\n",
    "    train_generator,\n",
    "    steps_per_epoch=len(train_generator),\n",
    "    epochs=epochs,\n",
    "    validation_data=val_generator,\n",
    "    validation_steps=len(val_generator),\n",
    "    callbacks=[early_stopping]  # Add the EarlyStopping callback here\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e416e66c-9a57-4611-b622-0ef53010ac2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the model\n",
    "\n",
    "model.save(model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ffde85b-1774-448a-9217-e2e1f159a2c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reload the model\n",
    "\n",
    "autoencoder_classifier = keras.models.load_model(model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "948aefce-556d-46f9-998a-44d1793f13a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test the model\n",
    "\n",
    "true_positives = 0  # Correctly identified as normal (valid images)\n",
    "false_negatives = 0  # Incorrectly identified as anomalous (valid images)\n",
    "true_negatives = 0  # Correctly identified as anomalous (invalid images)\n",
    "false_positives = 0  # Incorrectly identified as normal (invalid images)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1020482-be84-44fd-9f44-347c857a9c06",
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_image_dir = f'{target_dir}_images\\\\test\\\\valid'\n",
    "\n",
    "logger.iprint(\"Testing valid images\")\n",
    "# Recursively test with valid images (normal medical images)\n",
    "for root, dirs, files in os.walk(valid_image_dir):\n",
    "    for filename in files:\n",
    "        if filename.lower().endswith(('.png', '.jpg', '.jpeg')):\n",
    "            image_path = os.path.join(root, filename)\n",
    "            image = preprocess_image(image_path, img_width, img_height)\n",
    "\n",
    "            classifier_pred = autoencoder_classifier.predict(image)\n",
    "            logger.iprint(\"classifier_pred \" + str(classifier_pred))\n",
    "            predicted_class = (classifier_pred > threshold).astype(int)[0]\n",
    "\n",
    "            if predicted_class == 1:  # Correctly identified as normal\n",
    "                true_positives += 1\n",
    "                logger.iprint(\"True positive\")\n",
    "            else:  # Incorrectly identified as anomalous\n",
    "                false_negatives += 1\n",
    "                logger.eprint(\"Image is:\" + filename) \n",
    "                logger.eprint(\"False negative\")\n",
    "                display_fn_image(image_path)  # Display the image that was incorrectly classified as anomalous"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e180ffa-b2e3-41ec-8977-f64dcbd8a735",
   "metadata": {},
   "outputs": [],
   "source": [
    "logger.iprint(\"Testing invalid images\")\n",
    "\n",
    "invalid_image_dir = f'{target_dir}_images\\\\test\\\\invalid'\n",
    "# Recursively test with invalid images (anomalous or non-medical images)\n",
    "for root, dirs, files in os.walk(invalid_image_dir):\n",
    "    for filename in files:\n",
    "        if filename.lower().endswith(('.png', '.jpg', '.jpeg')):\n",
    "            image_path = os.path.join(root, filename)\n",
    "            image = preprocess_image(image_path, img_width, img_height)\n",
    "\n",
    "            # Predict the class of the image\n",
    "            classifier_pred = autoencoder_classifier.predict(image)\n",
    "            logger.iprint(\"classifier_pred\" + str(classifier_pred))\n",
    "            predicted_class = (classifier_pred > threshold).astype(int)[0]\n",
    "            \n",
    "            if predicted_class == 0:  # Correctly identified as anomalous\n",
    "                true_negatives += 1\n",
    "                logger.iprint(\"True negative\")\n",
    "            else:  # Incorrectly identified as normal\n",
    "                logger.eprint(\"Image is:\" + filename) \n",
    "                false_positives += 1\n",
    "                logger.eprint(\"False positive\")\n",
    "                # Optionally, display the image that was incorrectly classified as invalid\n",
    "                display_fp_image(image_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d681357-c15b-486b-8f31-4f7d60f7b4e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate performance metrics\n",
    "accuracy = (true_positives + true_negatives) / (true_positives + true_negatives + false_positives + false_negatives)\n",
    "precision = true_positives / (true_positives + false_positives) if (true_positives + false_positives) > 0 else 0\n",
    "recall = true_positives / (true_positives + false_negatives) if (true_positives + false_negatives) > 0 else 0\n",
    "f1_score = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0\n",
    "\n",
    "logger.iprint(f\"True Positives: {true_positives}\")\n",
    "logger.iprint(f\"False Positives: {false_positives}\")\n",
    "logger.iprint(f\"True Negatives: {true_negatives}\")\n",
    "logger.iprint(f\"False Negatives: {false_negatives}\")\n",
    "logger.iprint(f\"Accuracy: {accuracy:.4f}\")\n",
    "logger.iprint(f\"Precision: {precision:.4f}\")\n",
    "logger.iprint(f\"Recall: {recall:.4f}\")\n",
    "logger.iprint(f\"F1 Score: {f1_score:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "696fc4c2-757f-4e1e-aa18-81c2f7d06197",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"True Positives: {true_positives}\")\n",
    "print(f\"False Positives: {false_positives}\")\n",
    "print(f\"True Negatives: {true_negatives}\")\n",
    "print(f\"False Negatives: {false_negatives}\")\n",
    "print(f\"Accuracy: {accuracy:.4f}\")\n",
    "print(f\"Precision: {precision:.4f}\")\n",
    "print(f\"Recall: {recall:.4f}\")\n",
    "print(f\"F1 Score: {f1_score:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac2cb757-a24d-44ad-91f0-728e883b31f2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d64f4a12-fbbb-4ed5-87a7-926f2e60668f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "846bdbd8-3335-43d9-ba08-80bf3bef1bba",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21b1febf-a8a0-48c7-be6b-ec6bd8bd00b3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9ed7106-c136-4c08-9e55-a257b05d14e7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}