Inquiry Regarding Drift Correction Output in dme_example_colab.ipynb

[feizhao515]

Hello,

Thank you very much for your excellent work on drift correction for 3D SMLM data.

I am currently running the dme_example_colab.ipynb notebook and noticed the output message: "Rejecting step. Score: 1.374625." Could you kindly clarify if this indicates that the drift correction process failed for this dataset?

Thank you again for your assistance, and I look forward to your response.

[jcnossen]

Hi Feizhao, sorry for the slow reply. The "Rejecting step" does not indicate failure, it's just a message from the optimization algorithm. I implemented an gradient following algorithm that will try to take as large a step as possible, and whenever the score does not improve will lower the step size and try again. The optimization is considered done when smaller stepsizes also don't improve the score anymore (after 10 iterations with rejection)

[feizhao515]

Hello Jelmer, Thank you very much for your reply! I was wondering if you could kindly guide me on how to determine whether the algorithm has successfully find the correct drift for the data, or if it failed to identify the drift.

Thank you again for your help.

[jcnossen]

One approach is to split the dataset in 2 sections (like left and right parts of FOV) and see to what extend the drift matches.
Also I find that the initial RCC estimate isn't super reliable, so currently I prefer to disable it and just estimate using successively smaller values of framesPerBin, like this:

drift_dme2=ds.estimateDriftMinEntropy(framesPerBin=1000,initializeWithRCC=False,coarseFramesPerBin=10000, 
   coarseSigma=[1,1],estimatePrecision=False, maxspots=1000000, useCuda=True)

drift_dme3=ds.estimateDriftMinEntropy(framesPerBin=200,initializeWithRCC=False,initialEstimate=drift_dme2,
   estimatePrecision=False, maxspots=1000000, useCuda=True)

drift_dme4=ds.estimateDriftMinEntropy(framesPerBin=100,initializeWithRCC=False,initialEstimate=drift_dme3,
   estimatePrecision=False, maxspots=1000000, useCuda=True)

drift_dme4=ds.estimateDriftMinEntropy(framesPerBin=20,initializeWithRCC=False,initialEstimate=drift_dme4,
   estimatePrecision=False, maxspots=1000000, useCuda=True)

Hope that helps

[feizhao515]

Hello Jelmer, Thank you very much for your prompt reply! I couldn't locate the 'ds.estimateDriftMinEntropy' function which you mentioned in your reply in your code. Could you kindly guide me to where I can find it? Is it by chance the function 'dme_estimate' in the dme/dme.py? Thank you very much!