tutorial/4_hextof_workflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial for binning data from the HEXTOF instrument at FLASH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": "1"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "from typing import List\n",
    "from pathlib import Path\n",
    "import os\n",
    "\n",
    "from sed import SedProcessor\n",
    "from sed.dataset import dataset\n",
    "import xarray as xr\n",
    "\n",
    "%matplotlib widget\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get data paths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The paths are such that if you are on Maxwell, it uses those. Otherwise data is downloaded in current directory from Zenodo.\n",
    "\n",
    "Generally, if it is your beamtime, you can both read the raw data and write to processed directory. However, for the public data, you can not write to processed directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "beamtime_dir = \"/asap3/flash/gpfs/pg2/2023/data/11019101\" # on Maxwell\n",
    "if os.path.exists(beamtime_dir) and os.access(beamtime_dir, os.R_OK):\n",
    "    path = beamtime_dir + \"/raw/hdf/offline/fl1user3\"\n",
    "    meta_path = beamtime_dir + \"/shared\"\n",
    "    buffer_path = \"Gd_W110/processed/\"\n",
    "else:\n",
    "    # data_path can be defined and used to store the data in a specific location\n",
    "    dataset.get(\"Gd_W110\") # Put in Path to a storage of at least 10 GByte free space.\n",
    "    path = dataset.dir\n",
    "    meta_path = path\n",
    "    buffer_path = path + \"/processed/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "notebookRunGroups": {
     "groupValue": "1"
    }
   },
   "source": [
    "### Config setup\n",
    "Here we get the path to the config file and setup the relevant directories. This can also be done directly in the config file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": "1"
    }
   },
   "outputs": [],
   "source": [
    "# pick the default configuration file for hextof@FLASH\n",
    "config_file = Path('../src/sed/config/flash_example_config.yaml')\n",
    "assert config_file.exists()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The path to the processed folder can also be defined as a keyword argument later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# here we setup a dictionary that will be used to override the path configuration\n",
    "config_override = {\n",
    "    \"core\": {\n",
    "        \"paths\": {\n",
    "            \"raw\": path,\n",
    "            \"processed\": buffer_path,\n",
    "        },\n",
    "    },\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### cleanup previous config files\n",
    "In this notebook, we will show how calibration parameters can be generated. Therefore we want to clean the local directory of previously generated files.\n",
    "\n",
    "**WARNING** running the cell below will delete the \"sed_config.yaml\" file in the local directory. If these contain precious calibration parameters, **DO NOT RUN THIS CELL**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "local_folder_config = Path('./sed_config.yaml')\n",
    "if local_folder_config.exists():\n",
    "    os.remove(local_folder_config)\n",
    "    print(f'deleted local config file {local_folder_config}')\n",
    "assert not local_folder_config.exists()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load a chessy sample run\n",
    "The common starting point at a FLASH beamtime. Look at the Chessy sample!\n",
    "\n",
    "- run 44762: Chessy - FoV = 450 µm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate the Processor instance\n",
    "this cell generates an instance of the `SedProcessor` class. It will be our workhorse for the entire workflow.\n",
    "\n",
    "#### Important note\n",
    "The following extra arguments are available for FlashLoader. None of which are necessary to give but helpful to know.\n",
    "- **force_recreate**: Probably the most useful. In case the config is changed, this allows to reduce the raw h5 files to the the intermediate parquet format again. Otherwise, the schema between the saved dataframe and config differs.\n",
    "- **debug**: Setting this runs the reduction process in serial, so the errors are easier to find.\n",
    "- **remove_invalid_files**: Sometimes some critical channels defined in the config are missing in some raw files. Setting this will make sure to ignore such files.\n",
    "- **filter_timed_by_electron**: Defaults to True. When True, the timed dataframe will only\n",
    "                contain data points where valid electron events were detected. When False, all\n",
    "                timed data points are included regardless of electron detection (see https://github.com/OpenCOMPES/sed/issues/307)\n",
    "- **processed_dir**: Location to save the reduced parquet files. \n",
    "- **scicat_token**: Token from your scicat account.\n",
    "- **detector**: '1Q' and '4Q' detector for example. Useful when there are separate raw files for each detector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp = SedProcessor(runs=[44762], config=config_override, system_config=config_file, collect_metadata=False)\n",
    "# You can set collect_metadata=True if the scicat_url and scicat_token are defined"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add Jitter\n",
    "In order to avoid artifacts arising from incommensurate binning sizes with those imposed during data collection, e.g. by the detector, we jitter all the digital columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.add_jitter()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### inspect the dataframe\n",
    "Looking at the dataframe can give quick insight about the columns loaded and the data available. \n",
    "- `sp.dataframe` shows the structure of the dataframe without computing anything. Interesting here are the columns, and their type.\n",
    "- The `sp.dataframe.head()` function accesses the first 5 events in the dataframe, giving us a view of what the values of each column look like, without computing the whole thing.  `sp.dataframe.tail()`does the same from the end. \n",
    "- `sp.dataframe.compute()` will compute the whole dataframe, and can take a while. We should avoid doing this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.dataframe.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing event histograms\n",
    "For getting a first impression of the data, and to determine binning ranges, the method sp.`view_even_histogram()` allows visualizing the events in one dataframe partition as histograms. Default axes and ranges are defined in the config, and show the dldPosX, dldPosY, and dldTimeStep columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.view_event_histogram(dfpid=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Binning\n",
    "Here we define the parameters for binning the dataframe to an n-dimensional histogram, which we can then plot, analyze or save.\n",
    "\n",
    "If you never saw this before, the type after `:` is a \"hint\" to what type the object to the left will have. We include them here to make sure you know what each variable should be.\n",
    "```python\n",
    "a:int = 1 # a is an integer\n",
    "b:float = 1.0 # b is a float\n",
    "c:str = 1 # we hint c to be a string, but it is still an integer\n",
    "```\n",
    "This is totally optional, but can help you keep track of what you are doing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the name of the axes on which we want to bin\n",
    "axes: List[str] = ['dldPosY', 'dldPosX']\n",
    "# the number of bins for each axis\n",
    "bins: List[int] = [480, 480]\n",
    "# for each axis, the range of values to consider\n",
    "ranges: List[List[int]] = [[420,900], [420,900]]\n",
    "# here we compute the histogram\n",
    "res_chessy: xr.DataArray = sp.compute(bins=bins, axes=axes, ranges=ranges)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### visualize the result\n",
    "here we plot the binned histogram. The result is an xarray, which gives us some convenient visualization and simple plotting tools."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "res_chessy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "res_chessy.plot(robust=True) # robust is used to avoid outliers to dominate the color scale"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optical Spot Profile\n",
    "Here we load runs 44798 and 44799, which show the profile of the optical spot on the same spatial view as in our chessy run above. The two differ in transmission, being $T=1.0$ and $T=0.5$ respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp = SedProcessor(runs=[44798], config=config_override, system_config=config_file, collect_metadata=False)\n",
    "sp.add_jitter()\n",
    "res_t05: xr.DataArray = sp.compute(bins=bins, axes=axes, ranges=ranges)\n",
    "\n",
    "sp = SedProcessor(runs=[44799], config=config_override, system_config=config_file, collect_metadata=False)\n",
    "sp.add_jitter()\n",
    "res_t10: xr.DataArray = sp.compute(bins=bins, axes=axes, ranges=ranges)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig,ax = plt.subplots(1,3,figsize=(6,2), layout='tight')\n",
    "res_chessy.plot(ax=ax[0], robust=True, add_colorbar=False)\n",
    "res_t05.plot(ax=ax[1], robust=True, add_colorbar=False)\n",
    "res_t10.plot(ax=ax[2], robust=True, add_colorbar=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`TODO:` here we can add the evaluation of the spot size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Energy Calibration\n",
    "We now load a bias series, where the sample bias was varied, effectively shifting the energy spectra. This allows us to calibrate the conversion between the digital values of the dld and the energy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp = SedProcessor(runs=[44797], config=config_override, system_config=config_file, collect_metadata=False)\n",
    "sp.add_jitter()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `view_event_histogram()` function also to e.g. visualize the events per microbunch along the train, or hit multiplicity per microbunch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.view_event_histogram(dfpid=0, axes=[\"pulseId\", \"electronId\"], ranges=[[0, 600], [0,10]], bins=[100, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### sector alignment\n",
    "as usual first we jitter, but here we also align in time the 8 sectors of the dld. This is done by finding the time of the maximum of the signal in each sector, and then shifting the signal in each sector by the difference between the maximum time and the time of the maximum in each sector.\n",
    "\n",
    "For better precision, the photon peak can be used to track the energy shift."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.align_dld_sectors()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### time-of-flight spectrum\n",
    "to compare with what we see on the measurement computer, we might want to plot the time-of-flight spectrum. This is done here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.append_tof_ns_axis()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, to determine proper binning ranges, let's have again a look at the event histograms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.view_event_histogram(dfpid=0, axes=[\"sampleBias\", \"dldTime\"], ranges=[[27, 33], [650,1050]], bins=[50, 100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "axes = ['sampleBias','dldTime']\n",
    "bins = [5, 250]\n",
    "ranges = [[28,33],  [650,800]]\n",
    "res = sp.compute(bins=bins, axes=axes, ranges=ranges)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We binned not only in `dldTime` but also in `sampleBias`. This allows us to separate the spectra obtained at different bias values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "res.plot.line(x='dldTime'); # the ; here is to suppress an annoying output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### find calibration parameters\n",
    "We now will fit the tof-energy relation. This is done by finding the maxima of a peak in the tof spectrum, and then fitting the square root relation to obtain the calibration parameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "axes = ['sampleBias', 'dldTimeSteps']\n",
    "bins = [5, 500]\n",
    "ranges = [[28,33], [4000, 4800]]\n",
    "res = sp.compute(bins=bins, axes=axes, ranges=ranges)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.load_bias_series(binned_data=res)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ranges=(4120, 4200)\n",
    "ref_id=0\n",
    "sp.find_bias_peaks(ranges=ranges, ref_id=ref_id, apply=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.calibrate_energy_axis(\n",
    "    ref_energy=-.55,\n",
    "    method=\"lmfit\",\n",
    "    energy_scale='kinetic',\n",
    "    d={'value':1.0,'min': .2, 'max':1.0, 'vary':False},\n",
    "    t0={'value':5e-7, 'min': 1e-7, 'max': 1e-6, 'vary':True},\n",
    "    E0={'value': 0., 'min': -100, 'max': 100, 'vary': True},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### generate the energy axis\n",
    "Now that we have the calibration parameters, we can generate the energy axis for each spectrum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.append_energy_axis()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets have a look at the dataset, and the columns we added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.dataframe[['dldTime','dldTimeSteps','energy','dldSectorID']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bin in energy\n",
    "With the newly added column, we can now bin directly in energy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "axes: List[str] = ['sampleBias', 'energy']\n",
    "bins: List[int] = [5, 500]\n",
    "ranges: List[List[int]] = [[28,33], [-10,10]]\n",
    "res: xr.DataArray = sp.compute(bins=bins, axes=axes, ranges=ranges)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure() # if you are using interactive plots, you'll need to generate a new figure explicitly every time.\n",
    "res.mean('sampleBias').plot.line(x='energy',linewidth=3)\n",
    "res.plot.line(x='energy',linewidth=1,alpha=.5);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### correct offsets\n",
    "The energy axis is now correct, taking the sample bias of the measurement into account. Additionally, we can compensate the photon energy (`monochromatorPhotonEnergy`) and the `tofVoltage`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.add_energy_offset(\n",
    "    columns=['monochromatorPhotonEnergy','tofVoltage'],\n",
    "    weights=[-1,-1],\n",
    "    preserve_mean=[True, True],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we bin again and see the result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "axes = ['sampleBias', 'energy']\n",
    "bins = [5, 500]\n",
    "ranges = [[28,33], [-10,2]]\n",
    "res = sp.compute(bins=bins, axes=axes, ranges=ranges)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "ax = plt.subplot(111)\n",
    "res.energy.attrs['unit'] = 'eV' # add units to the axes\n",
    "res.mean('sampleBias').plot.line(x='energy',linewidth=3, ax=ax)\n",
    "res.plot.line(x='energy',linewidth=1,alpha=.5,label='all',ax=ax);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### save the calibration parameters\n",
    "The parameters we have found can be saved to a file, so that we can use them later. This means the calibration can be used for different runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.save_energy_calibration()\n",
    "sp.save_energy_offset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A more general function, which saves parameters for all the calibrations performed. Use either the above or below function. They are equivalent (and overwrite each other)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.save_workflow_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Correct delay axis\n",
    "To calibrate the pump-probe delay axis, we need to shift the delay stage values to center the pump-probe-time overlap `time zero`.\n",
    "Also, we want to correct the SASE jitter, using information from the `bam` column.\n",
    "\n",
    "Here we load multiple runs at once"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": "2"
    }
   },
   "outputs": [],
   "source": [
    "sp = SedProcessor(\n",
    "    runs=[44824,44825,44826,44827],\n",
    "    config=config_override,\n",
    "    system_config=config_file,\n",
    "    collect_metadata=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the workflow from the config file\n",
    "as we have saved some calibration and correction parameters, we can now run the workflow from the config file. This is done by calling each of the correction functions, with no parameters. The functions will then load the parameters from the config file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": "2"
    }
   },
   "outputs": [],
   "source": [
    "sp.add_jitter()\n",
    "sp.align_dld_sectors()\n",
    "sp.append_energy_axis()\n",
    "sp.add_energy_offset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### plot the delayStage values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "axes = ['energy','delayStage']\n",
    "bins = [100,150]\n",
    "delay_start,delay_stop=1462.00,1464.85\n",
    "ranges = [[-5,2], [delay_start,delay_stop]]\n",
    "res = sp.compute(bins=bins, axes=axes, ranges=ranges)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig,ax = plt.subplots(1,2,figsize=(6,2.25), layout='constrained')\n",
    "res.plot(robust=True, ax=ax[0])\n",
    "bg = res.isel(delayStage=slice(0,10)).mean('delayStage')\n",
    "(res-bg).plot(robust=True, ax=ax[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": "2"
    }
   },
   "outputs": [],
   "source": [
    "sp.add_delay_offset(\n",
    "    constant=-1463.7, # this is time zero\n",
    "    flip_delay_axis=True, # invert the direction of the delay axis\n",
    "    columns=['bam'], # use the bam to offset the values\n",
    "    weights=[-0.001], # bam is in fs, delay in ps\n",
    "    preserve_mean=True # preserve the mean of the delay axis\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.dataframe # This has generated too many layers, there is room for improvement!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### bin in the corrected delay axis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": "2"
    }
   },
   "outputs": [],
   "source": [
    "axes = ['energy','delayStage']\n",
    "bins = [100,150]\n",
    "delay_start,delay_stop=1462.00,1464.85\n",
    "ranges = [[-3,2], [-1.1, 1.75]]\n",
    "res = sp.compute(bins=bins, axes=axes, ranges=ranges)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": "2"
    }
   },
   "outputs": [],
   "source": [
    "fig,ax = plt.subplots(1,2,figsize=(6,2.25))\n",
    "res.plot(robust=True, ax=ax[0])\n",
    "bg = res.sel(delayStage=slice(-1,-0.2)).mean('delayStage')\n",
    "(res-bg).plot(robust=True, ax=ax[1])\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may note some intensity variation along the delay axis. This comes mainly from inhomogeneous speed of the delay stage, and thus inequivalent amounts of time spent on every delay point. This can be corrected for by normalizing the data to the acquisition time per delay point:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = sp.compute(bins=bins, axes=axes, ranges=ranges, normalize_to_acquisition_time=\"delayStage\")\n",
    "fig,ax = plt.subplots(1,2,figsize=(6,2.25), layout='constrained')\n",
    "res.plot(robust=True, ax=ax[0])\n",
    "bg = res.sel(delayStage=slice(-1,-.2)).mean('delayStage')\n",
    "(res-bg).plot(robust=True, ax=ax[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### save parameters\n",
    "as before, we can save the parameters we just used in the config for the next run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": "2"
    }
   },
   "outputs": [],
   "source": [
    "sp.save_delay_offsets()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run workflow entirely from config.\n",
    "Once all the calibrations are done, a new run can be loaded by simply calling all the calibration functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sed.core.config import load_config\n",
    "import numpy as np\n",
    "metadata = load_config(meta_path + \"/44824_20230324T060430.json\")\n",
    "\n",
    "# Fix metadata\n",
    "metadata[\"scientificMetadata\"][\"Laser\"][\"wavelength\"][\"value\"] = float(metadata[\"scientificMetadata\"][\"Laser\"][\"wavelength\"][\"value\"][:-2])\n",
    "metadata[\"scientificMetadata\"][\"Laser\"][\"energy\"] = {\"value\": 1239.84/metadata[\"scientificMetadata\"][\"Laser\"][\"wavelength\"][\"value\"], \"unit\": \"eV\"}\n",
    "metadata[\"scientificMetadata\"][\"Laser\"][\"polarization\"] = [1, 1, 0, 0]\n",
    "metadata[\"scientificMetadata\"][\"Collection\"][\"field_aperture_x\"] = float(metadata[\"scientificMetadata\"][\"Collection\"][\"field_aperture_x\"])\n",
    "metadata[\"scientificMetadata\"][\"Collection\"][\"field_aperture_y\"] = float(metadata[\"scientificMetadata\"][\"Collection\"][\"field_aperture_y\"])\n",
    "metadata[\"pi\"] = {\"institute\": \"JGU Mainz\"}\n",
    "metadata[\"proposer\"] = {\"institute\": \"TU Dortmund\"}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": ""
    }
   },
   "outputs": [],
   "source": [
    "sp = SedProcessor(\n",
    "    runs=[44824,44825,44826,44827],\n",
    "    config=config_override,\n",
    "    system_config=config_file,\n",
    "    metadata = metadata,\n",
    "    collect_metadata=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": ""
    }
   },
   "outputs": [],
   "source": [
    "sp.add_jitter()\n",
    "sp.align_dld_sectors()\n",
    "sp.append_tof_ns_axis()\n",
    "sp.append_energy_axis()\n",
    "sp.add_energy_offset()\n",
    "sp.add_delay_offset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "axes = ['energy','delayStage']\n",
    "bins = [100,150]\n",
    "delay_start,delay_stop=1462.00,1464.85\n",
    "ranges = [[-5,2], [-1.1, 1.75]]\n",
    "res = sp.compute(bins=bins, axes=axes, ranges=ranges, normalize_to_acquisition_time=\"delayStage\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig,ax = plt.subplots(1,2,figsize=(6,2.25), layout='constrained')\n",
    "res.plot(robust=True, ax=ax[0])\n",
    "bg = res.sel(delayStage=slice(-1,-.2)).mean('delayStage')\n",
    "(res-bg).plot(robust=True, ax=ax[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save results\n",
    "binned data can now be saved as h5 or tiff. igor binaries soon to come if requested!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.save('runs44824-27.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.save('runs44824-27.tiff')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.save(\"runs44824-27.nxs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}