brainstorming on upgrading ephys pipeline to latest version (0.4*)

[ttngu207]

Notes from a call with @tabedzki to brainstorm on upgrading ephys pipeline for BrainCOGS to the latest version

1. PR to update kilosort "core_files"
2. new branch named new_dj on braincogs fork
3. PR main_external_storage (from upstream) into this new branch new_dj
4. make the changes on blob@ephys_store (for LFP and waveform) same as the current braincogs main

For production / migration

1. rename the current ephys schema to ephys_old
2. update the dependency (in pyproject.toml) (point to the new_dj branch)
3. instantiate the pipeline again

• add ephys-processed store into dj.config['stores'] (probably the exact same definition as ephys_store

4. write a data-copy script to copy all data from the ephys_old schema into the new ephys schema

• for each table ephys.EphysRecording.insert(old_ephys.EphysRecording, ignore_extra_fields=True)

        "ephys-processed": {
            "protocol": "file",
            "location": "/Volumes/u19_dj/external_dj_blobs",
            "stage": "/Volumes/u19_dj/external_dj_blobs"
        }