Sporadic error when updating `hparams`

[SimWdm]

Hi Simon,

Lately, I was using DDW again extensively and came across this "sporadic" error again... only for some of the many jobs but they are still popping up. I double checked with @rdrighetto and we have the latest version.

TypeError: 'NoneType' object does not support item assignment

I will paste the full error message below:

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/ddw/fit_model │
│ .py:273 in fit_model                                                         │
│                                                                              │
│   270 │   # fit the model                                                    │
│   271 │   if val_data_exists and resume_from_checkpoint is None:             │
│   272 │   │   trainer.validate(lit_unet, val_dataloader)                     │
│ ❱ 273 │   trainer.fit(                                                       │
│   274 │   │   #ckpt_path=resume_from_checkpoint,  # for pytorch-lightning >= │
│   275 │   │   model=lit_unet,                                                │
│   276 │   │   train_dataloaders=fitting_dataloader,                          │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/trainer/trainer.py:579 in fit                                           │
│                                                                              │
│    576 │   │   if not isinstance(model, pl.LightningModule):                 │
│    577 │   │   │   raise TypeError(f"`Trainer.fit()` requires a `LightningMo │
│    578 │   │   self.strategy._lightning_module = model                       │
│ ❱  579 │   │   call._call_and_handle_interrupt(                              │
│    580 │   │   │   self, self._fit_impl, model, train_dataloaders, val_datal │
│    581 │   │   )                                                             │
│    582                                                                       │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/trainer/call.py:38 in _call_and_handle_interrupt                        │
│                                                                              │
│   35 │   │   if trainer.strategy.launcher is not None:                       │
│   36 │   │   │   return trainer.strategy.launcher.launch(trainer_fn, *args,  │
│   37 │   │   else:                                                           │
│ ❱ 38 │   │   │   return trainer_fn(*args, **kwargs)                          │
│   39 │                                                                       │
│   40 │   except _TunerExitException:                                         │
│   41 │   │   trainer._call_teardown_hook()                                   │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/trainer/trainer.py:621 in _fit_impl                                     │
│                                                                              │
│    618 │   │   │   model_provided=True,                                      │
│    619 │   │   │   model_connected=self.lightning_module is not None,        │
│    620 │   │   )                                                             │
│ ❱  621 │   │   self._run(model, ckpt_path=self.ckpt_path)                    │
│    622 │   │                                                                 │
│    623 │   │   assert self.state.stopped                                     │
│    624 │   │   self.training = False                                         │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/trainer/trainer.py:1058 in _run                                         │
│                                                                              │
│   1055 │   │                                                                 │
│   1056 │   │   self._checkpoint_connector.resume_end()                       │
│   1057 │   │                                                                 │
│ ❱ 1058 │   │   results = self._run_stage()                                   │
│   1059 │   │                                                                 │
│   1060 │   │   log.detail(f"{self.__class__.__name__}: trainer tearing down" │
│   1061 │   │   self._teardown()                                              │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/trainer/trainer.py:1137 in _run_stage                                   │
│                                                                              │
│   1134 │   │   │   return self._run_evaluate()                               │
│   1135 │   │   if self.predicting:                                           │
│   1136 │   │   │   return self._run_predict()                                │
│ ❱ 1137 │   │   self._run_train()                                             │
│   1138 │                                                                     │
│   1139 │   def _pre_training_routine(self) -> None:                          │
│   1140 │   │   # wait for all to join if on distributed                      │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/trainer/trainer.py:1160 in _run_train                                   │
│                                                                              │
│   1157 │   │   self.fit_loop.trainer = self                                  │
│   1158 │   │                                                                 │
│   1159 │   │   with torch.autograd.set_detect_anomaly(self._detect_anomaly): │
│ ❱ 1160 │   │   │   self.fit_loop.run()                                       │
│   1161 │                                                                     │
│   1162 │   def _run_evaluate(self) -> _EVALUATE_OUTPUT:                      │
│   1163 │   │   assert self.evaluating                                        │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/loops/loop.py:200 in run                                                │
│                                                                              │
│   197 │   │   │   try:                                                       │
│   198 │   │   │   │   self.on_advance_start(*args, **kwargs)                 │
│   199 │   │   │   │   self.advance(*args, **kwargs)                          │
│ ❱ 200 │   │   │   │   self.on_advance_end()                                  │
│   201 │   │   │   │   self._restarting = False                               │
│   202 │   │   │   except StopIteration:                                      │
│   203 │   │   │   │   break                                                  │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/loops/fit_loop.py:296 in on_advance_end                                 │
│                                                                              │
│   293 │   │                                                                  │
│   294 │   │   # call train epoch end hooks                                   │
│   295 │   │   self.trainer._call_callback_hooks("on_train_epoch_end")        │
│ ❱ 296 │   │   self.trainer._call_lightning_module_hook("on_train_epoch_end") │
│   297 │   │                                                                  │
│   298 │   │   self.trainer._logger_connector.on_epoch_end()                  │
│   299                                                                        │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/trainer/trainer.py:1302 in _call_lightning_module_hook                  │
│                                                                              │
│   1299 │   │   pl_module._current_fx_name = hook_name                        │
│   1300 │   │                                                                 │
│   1301 │   │   with self.profiler.profile(f"[LightningModule]{pl_module.__cl │
│ ❱ 1302 │   │   │   output = fn(*args, **kwargs)                              │
│   1303 │   │                                                                 │
│   1304 │   │   # restore current_fx when nested context                      │
│   1305 │   │   pl_module._current_fx_name = prev_fx_name                     │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/ddw/utils/une │
│ t.py:85 in on_train_epoch_end                                                │
│                                                                              │
│    82 │   │   │   self.current_epoch + 1                                     │
│    83 │   │   ) % self.update_subtomo_missing_wedges_every_n_epochs == 0:  # │
│    84 │   │   │   self.update_subtomo_missing_wedges()                       │
│ ❱  85 │   │   │   self.update_normalization()                                │
│    86 │                                                                      │
│    87 │   def configure_optimizers(self):                                    │
│    88 │   │   optimizer = torch.optim.Adam(self.parameters(), **self.adam_pa │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/ddw/utils/une │
│ t.py:166 in update_normalization                                             │
│                                                                              │
│   163 │   │   # update normalization in hparams                              │
│   164 │   │   self.unet_params["normalization_loc"] = loc                    │
│   165 │   │   self.unet_params["normalization_scale"] = scale                │
│ ❱ 166 │   │   self.update_hparam("unet_params", self.unet_params)            │
│   167 │   │   self.log("normalization/loc", loc)                             │
│   168 │   │   self.log("normalization/scale", scale)                         │
│   169                                                                        │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/ddw/utils/une │
│ t.py:178 in update_hparam                                                    │
│                                                                              │
│   175 │   │   logdir = f"{logger.save_dir}/{logger.name}/version_{logger.ver │
│   176 │   │   hparams_file = f"{logdir}/hparams.yaml"                        │
│   177 │   │   hparams = yaml.safe_load(open(hparams_file, "r"))              │
│ ❱ 178 │   │   hparams[hparam] = value                                        │
│   179 │   │   with open(hparams_file, "w") as f:                             │
│   180 │   │   │   yaml.dump(hparams, f)                                      │
│   181                                                                        │
╰──────────────────────────────────────────────────────────────────────────────╯
TypeError: 'NoneType' object does not support item assignment
Command exited with non-zero status 1
45074.13user 2356.74system 1:47:50elapsed 733%CPU (0avgtext+0avgdata 3734392maxresident)k
0inputs+98117120outputs (136948422major+146981054minor)pagefaults 0swaps
srun: error: sge01: task 1: Exited with exit code 1
[rank0]:[E ProcessGroupNCCL.cpp:523] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=32728, OpType=ALLREDUCE, NumelIn=276641, NumelOut=276641, Timeout(ms)=1800000) ran for 1800387 milliseconds before timing out.
[rank0]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank0]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[rank0]:[E ProcessGroupNCCL.cpp:1182] [Rank 0] NCCL watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=32728, OpType=ALLREDUCE, NumelIn=276641, NumelOut=276641, Timeout(ms)=1800000) ran for 1800387 milliseconds before timing out.
Exception raised from checkTimeout at /opt/conda/conda-bld/pytorch_1711403380909/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x1461aaf80d87 in /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3_python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1e6 (0x14615f2384d6 in /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3_python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x14615f23ba2d in /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3_python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x14615f23c629 in /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3_python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdbbf4 (0x1461aaac7bf4 in /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3_python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #5: <unknown function> + 0x94ac3 (0x1461b333eac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: <unknown function> + 0x126850 (0x1461b33d0850 in /lib/x86_64-linux-gnu/libc.so.6)

Command terminated by signal 6
5867.11user 1972.00system 2:17:12elapsed 95%CPU (0avgtext+0avgdata 3658520maxresident)k
368inputs+98025688outputs (49378770major+75166635minor)pagefaults 0swaps

Originally posted by @Phaips in #8

[SimWdm]

Judging from the error message TypeError: 'NoneType' object does not support item assignment upon hparams[hparam] value, it seems like hparams is None, which could be due to a corrupt hyper-parameter file.

If you still have the logs of such a failed run, could you check the hparams.yaml file within the version_X directory of the run and post the contents here?

[Phaips]

Hi Simon,

Thanks for looking in to this so quickly! The files seem to look the same...
A failed hparams.yaml:

adam_params:
  lr: 0.0004
subtomo_dir: out/subtomos
unet_params:
  chans: 32
  drop_prob: 0.0
  num_downsample_layers: 3
update_subtomo_missing_wedges_every_n_epochs: 10

a "successful" hparams.yml:

adam_params:
  lr: 0.0004
subtomo_dir: out/subtomos
unet_params:
  chans: 32
  drop_prob: 0.0
  num_downsample_layers: 3
update_subtomo_missing_wedges_every_n_epochs: 10

[SimWdm]

Oh wow that makes things tricky!

As the error occurs at random, I think we should first try to catch it.

Can you run ddw directly on a server without SLURM or anything? If so, you could wrap the

hparams_file = f"{logdir}/hparams.yaml"
hparams = yaml.safe_load(open(hparams_file, "r"))
hparams[hparam] = value

code block starting here

DeepDeWedge/ddw/utils/unet.py

Line 176 in 3ff5721

hparams_file = f"{logdir}/hparams.yaml"

with a conditional debugger trace like this

try:
    hparams_file = f"{logdir}/hparams.yaml"
    hparams = yaml.safe_load(open(hparams_file, "r"))
    hparams[hparam] = value
except:
    import pdb; pdb.set_trace()

Don't forget to re-install afterwards.

Then you can start a run and if the hparam update fails, you will be sent to the debugger and can inspect the hparams and value objects. Once you have done this we will probably have a much clearer picture of what's going on and how we can prevent the error from happening.

[Phaips]

Hey Simon,

Sorry for the slow reply. Running the jobs took quite a bit of time. I ran it twice on two different tomograms and both jobs finished without any error so the try block was never excepted...
In the meantime a job submitted via our usual SLURM submission did fail but I am not sure pdb works properly when not run in an active terminal window? Anyway I will paste below the error:

Epoch 259: 100%|██████████| 37/37 [00:18<00:00,  2.01it/s, loss=33.7, v_num=0, fitting_loss=33.80, val_loss=36.30]> /scicore/projects/scicore-p-structsoft/ubuntu/software/DeepDeWedge/ddw/utils/unet.py(183)update_hparam()
-> hparams = yaml.safe_load(open(hparams_file, "r"))
(Pdb) 
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/DeepDeWedge/ddw/fit_m │
│ odel.py:273 in fit_model                                                     │
│                                                                              │
│   270 │   # fit the model                                                    │
│   271 │   if val_data_exists and resume_from_checkpoint is None:             │
│   272 │   │   trainer.validate(lit_unet, val_dataloader)                     │
│ ❱ 273 │   trainer.fit(                                                       │
│   274 │   │   #ckpt_path=resume_from_checkpoint,  # for pytorch-lightning >= │
│   275 │   │   model=lit_unet,                                                │
│   276 │   │   train_dataloaders=fitting_dataloader,                          │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/trainer/trainer.py:579 in fit                                           │
│                                                                              │
│    576 │   │   if not isinstance(model, pl.LightningModule):                 │
│    577 │   │   │   raise TypeError(f"`Trainer.fit()` requires a `LightningMo │
│    578 │   │   self.strategy._lightning_module = model                       │
│ ❱  579 │   │   call._call_and_handle_interrupt(                              │
│    580 │   │   │   self, self._fit_impl, model, train_dataloaders, val_datal │
│    581 │   │   )                                                             │
│    582                                                                       │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/trainer/call.py:38 in _call_and_handle_interrupt                        │
│                                                                              │
│   35 │   │   if trainer.strategy.launcher is not None:                       │
│   36 │   │   │   return trainer.strategy.launcher.launch(trainer_fn, *args,  │
│   37 │   │   else:                                                           │
│ ❱ 38 │   │   │   return trainer_fn(*args, **kwargs)                          │
│   39 │                                                                       │
│   40 │   except _TunerExitException:                                         │
│   41 │   │   trainer._call_teardown_hook()                                   │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/trainer/trainer.py:621 in _fit_impl                                     │
│                                                                              │
│    618 │   │   │   model_provided=True,                                      │
│    619 │   │   │   model_connected=self.lightning_module is not None,        │
│    620 │   │   )                                                             │
│ ❱  621 │   │   self._run(model, ckpt_path=self.ckpt_path)                    │
│    622 │   │                                                                 │
│    623 │   │   assert self.state.stopped                                     │
│    624 │   │   self.training = False                                         │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/trainer/trainer.py:1058 in _run                                         │
│                                                                              │
│   1055 │   │                                                                 │
│   1056 │   │   self._checkpoint_connector.resume_end()                       │
│   1057 │   │                                                                 │
│ ❱ 1058 │   │   results = self._run_stage()                                   │
│   1059 │   │                                                                 │
│   1060 │   │   log.detail(f"{self.__class__.__name__}: trainer tearing down" │
│   1061 │   │   self._teardown()                                              │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/trainer/trainer.py:1137 in _run_stage                                   │
│                                                                              │
│   1134 │   │   │   return self._run_evaluate()                               │
│   1135 │   │   if self.predicting:                                           │
│   1136 │   │   │   return self._run_predict()                                │
│ ❱ 1137 │   │   self._run_train()                                             │
│   1138 │                                                                     │
│   1139 │   def _pre_training_routine(self) -> None:                          │
│   1140 │   │   # wait for all to join if on distributed                      │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/trainer/trainer.py:1160 in _run_train                                   │
│                                                                              │
│   1157 │   │   self.fit_loop.trainer = self                                  │
│   1158 │   │                                                                 │
│   1159 │   │   with torch.autograd.set_detect_anomaly(self._detect_anomaly): │
│ ❱ 1160 │   │   │   self.fit_loop.run()                                       │
│   1161 │                                                                     │
│   1162 │   def _run_evaluate(self) -> _EVALUATE_OUTPUT:                      │
│   1163 │   │   assert self.evaluating                                        │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/loops/loop.py:200 in run                                                │
│                                                                              │
│   197 │   │   │   try:                                                       │
│   198 │   │   │   │   self.on_advance_start(*args, **kwargs)                 │
│   199 │   │   │   │   self.advance(*args, **kwargs)                          │
│ ❱ 200 │   │   │   │   self.on_advance_end()                                  │
│   201 │   │   │   │   self._restarting = False                               │
│   202 │   │   │   except StopIteration:                                      │
│   203 │   │   │   │   break                                                  │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/loops/fit_loop.py:296 in on_advance_end                                 │
│                                                                              │
│   293 │   │                                                                  │
│   294 │   │   # call train epoch end hooks                                   │
│   295 │   │   self.trainer._call_callback_hooks("on_train_epoch_end")        │
│ ❱ 296 │   │   self.trainer._call_lightning_module_hook("on_train_epoch_end") │
│   297 │   │                                                                  │
│   298 │   │   self.trainer._logger_connector.on_epoch_end()                  │
│   299                                                                        │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/pytorch_light │
│ ning/trainer/trainer.py:1302 in _call_lightning_module_hook                  │
│                                                                              │
│   1299 │   │   pl_module._current_fx_name = hook_name                        │
│   1300 │   │                                                                 │
│   1301 │   │   with self.profiler.profile(f"[LightningModule]{pl_module.__cl │
│ ❱ 1302 │   │   │   output = fn(*args, **kwargs)                              │
│   1303 │   │                                                                 │
│   1304 │   │   # restore current_fx when nested context                      │
│   1305 │   │   pl_module._current_fx_name = prev_fx_name                     │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/DeepDeWedge/ddw/utils │
│ /unet.py:85 in on_train_epoch_end                                            │
│                                                                              │
│    82 │   │   │   self.current_epoch + 1                                     │
│    83 │   │   ) % self.update_subtomo_missing_wedges_every_n_epochs == 0:  # │
│    84 │   │   │   self.update_subtomo_missing_wedges()                       │
│ ❱  85 │   │   │   self.update_normalization()                                │
│    86 │                                                                      │
│    87 │   def configure_optimizers(self):                                    │
│    88 │   │   optimizer = torch.optim.Adam(self.parameters(), **self.adam_pa │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/DeepDeWedge/ddw/utils │
│ /unet.py:166 in update_normalization                                         │
│                                                                              │
│   163 │   │   # update normalization in hparams                              │
│   164 │   │   self.unet_params["normalization_loc"] = loc                    │
│   165 │   │   self.unet_params["normalization_scale"] = scale                │
│ ❱ 166 │   │   self.update_hparam("unet_params", self.unet_params)            │
│   167 │   │   self.log("normalization/loc", loc)                             │
│   168 │   │   self.log("normalization/scale", scale)                         │
│   169                                                                        │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/DeepDeWedge/ddw/utils │
│ /unet.py:183 in update_hparam                                                │
│                                                                              │
│   180 │   │   │   hparams[hparam] = value                                    │
│   181 │   │   except:                                                        │
│   182 │   │   │   import pdb; pdb.set_trace()                                │
│ ❱ 183 │   │   hparams = yaml.safe_load(open(hparams_file, "r"))              │
│   184 │   │   hparams[hparam] = value                                        │
│   185 │   │   with open(hparams_file, "w") as f:                             │
│   186 │   │   │   yaml.dump(hparams, f)                                      │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/DeepDeWedge/ddw/utils │
│ /unet.py:183 in update_hparam                                                │
│                                                                              │
│   180 │   │   │   hparams[hparam] = value                                    │
│   181 │   │   except:                                                        │
│   182 │   │   │   import pdb; pdb.set_trace()                                │
│ ❱ 183 │   │   hparams = yaml.safe_load(open(hparams_file, "r"))              │
│   184 │   │   hparams[hparam] = value                                        │
│   185 │   │   with open(hparams_file, "w") as f:                             │
│   186 │   │   │   yaml.dump(hparams, f)                                      │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/bdb.py:90 in trace_dispatch │
│                                                                              │
│    87 │   │   if self.quitting:                                              │
│    88 │   │   │   return # None                                              │
│    89 │   │   if event == 'line':                                            │
│ ❱  90 │   │   │   return self.dispatch_line(frame)                           │
│    91 │   │   if event == 'call':                                            │
│    92 │   │   │   return self.dispatch_call(frame, arg)                      │
│    93 │   │   if event == 'return':                                          │
│                                                                              │
│ /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3 │
│ _python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/bdb.py:115 in dispatch_line │
│                                                                              │
│   112 │   │   """                                                            │
│   113 │   │   if self.stop_here(frame) or self.break_here(frame):            │
│   114 │   │   │   self.user_line(frame)                                      │
│ ❱ 115 │   │   │   if self.quitting: raise BdbQuit                            │
│   116 │   │   return self.trace_dispatch                                     │
│   117 │                                                                      │
│   118 │   def dispatch_call(self, frame, arg):                               │
╰──────────────────────────────────────────────────────────────────────────────╯
BdbQuit
Command exited with non-zero status 1
42891.97user 968.72system 1:34:18elapsed 775%CPU (0avgtext+0avgdata 3507980maxresident)k
0inputs+77733776outputs (107910691major+130596470minor)pagefaults 0swaps
Epoch 259: 100%|██████████| 37/37 [01:31<00:00,  2.47s/it, loss=33.7, v_num=0, fitting_loss=33.80, val_loss=36.30]srun: error: sgc05: task 0: Exited with exit code 1
[rank1]:[E ProcessGroupNCCL.cpp:523] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=26468, OpType=ALLREDUCE, NumelIn=276641, NumelOut=276641, Timeout(ms)=1800000) ran for 1800663 milliseconds before timing out.
[rank1]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank1]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[rank1]:[E ProcessGroupNCCL.cpp:1182] [Rank 1] NCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=26468, OpType=ALLREDUCE, NumelIn=276641, NumelOut=276641, Timeout(ms)=1800000) ran for 1800663 milliseconds before timing out.
Exception raised from checkTimeout at /opt/conda/conda-bld/pytorch_1711403380909/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x14cfd5980d87 in /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3_python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1e6 (0x14cf89c384d6 in /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3_python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x14cf89c3ba2d in /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3_python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x14cf89c3c629 in /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3_python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdbbf4 (0x14cfd54c7bf4 in /scicore/projects/scicore-p-structsoft/ubuntu/software/Miniconda3/miniconda3_python3.12.1/envs/DeepDeWedge-ub/lib/python3.10/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #5: <unknown function> + 0x94ac3 (0x14cfddedaac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: <unknown function> + 0x126850 (0x14cfddf6c850 in /lib/x86_64-linux-gnu/libc.so.6)

At this point I am not sure if the problem could be coming from SLURM. I am not experienced with SLURM but I guess it could be that especially in HPC environment one can run into problems with i.e. file access timing and in our case with the hparams.yaml. When running under SLURM, there might be moments when the file is being read/written by multiple processes simultaneously?

Claude 3.7 Sonnet recommended a solution trying to catch such errors like:

def update_hparam(self, hparam, value):
    logger = self.trainer.logger
    if logger is None or not hasattr(logger, 'save_dir') or logger.save_dir is None:
        print("Warning: Logger not available or properly configured, skipping hparam update")
        return
        
    try:
        logdir = f"{logger.save_dir}/{logger.name}/version_{logger.version}"
        hparams_file = f"{logdir}/hparams.yaml"
        
        # Debug info
        import os
        print(f"Updating hparam '{hparam}' in file: {hparams_file}")
        print(f"File exists: {os.path.exists(hparams_file)}")
        
        if os.path.exists(hparams_file):
            # Read file content for debugging
            with open(hparams_file, "r") as f:
                content = f.read()
                print(f"File content length: {len(content)}")
                print(f"File content preview: {content[:100]}...")
                
            # Load with explicit error handling
            try:
                hparams = yaml.safe_load(content) if content.strip() else {}
            except Exception as e:
                print(f"YAML parsing error: {e}")
                hparams = {}
        else:
            print(f"Creating new hparams file at {hparams_file}")
            hparams = {}
        
        # Add defensive check
        if hparams is None:
            print("Warning: hparams loaded as None, initializing empty dict")
            hparams = {}
            
        # Debug before assignment
        print(f"Current hparams: {hparams}")
        print(f"Setting {hparam} to {value}")
        
        # Update
        hparams[hparam] = value
        
        # Write back with extra safety
        temp_file = f"{hparams_file}.tmp"
        with open(temp_file, "w") as f:
            yaml.dump(hparams, f)
        
        # Atomic replacement
        os.replace(temp_file, hparams_file)
        print(f"Successfully updated hparams file")
        
    except Exception as e:
        import traceback
        print(f"Error in update_hparam: {e}")
        print(traceback.format_exc())
        # Continue without failing

but I am not sure if this will fix it and also if it would impact the performance of the model(?)
I will try run DDW a couple of more times from an active terminal to see if I can catch this error with the debugging and would let you know in case.
Thanks!

[rdrighetto]

At this point I am not sure if the problem could be coming from SLURM. I am not experienced with SLURM but I guess it could be that especially in HPC environment one can run into problems with i.e. file access timing and in our case with the hparams.yaml. When running under SLURM, there might be moments when the file is being read/written by multiple processes simultaneously?

It's true that on parallel filesystems like ours things can get unstable when intensive I/O like this is happening. I remember testing running DDW entirely on the local SSD (scratch location defined by $TMPDIR), copying the input data and everything, but the "random error" (at least the previous one) still occurred. This time might be something different though and we should perhaps test again?

[SimWdm]

Hi @Phaips and @rdrighetto ,

thanks for looking into this further!

Unfortunately, I don't have much experience with SLURM since we don't have a SLURM cluster in our research group, which is also why I always run ddw directly on a specific server, so I can't test whether the issue is SLURM-related or or not. Sorry for that! 😕

I remember testing running DDW entirely on the local SSD (scratch location defined by $TMPDIR), copying the input data and everything, but the "random error" (at least the previous one) still occurred. This time might be something different though and we should perhaps test again?

I think the hparams error is different than the previous error which always occured during loading the mrcfiles, but the underlying issue could be the same: Some file (be it a subtomo .mrc file or the hparams.yaml) cannot be read properly from disk.

As we have not yet found a pattern our only chance to find the root of all evil is probably to start jobs over and over until we get useful error messages. To this end, I have created a new branch which hopefully catches the error so we can see what is actually the problem:

    def update_hparam(self, hparam, value):
        """
        Update a hyperparameter in the hparams.yaml file.
        """
        logger = self.trainer.logger
        logdir = f"{logger.save_dir}/{logger.name}/version_{logger.version}"
        hparams_file = f"{logdir}/hparams.yaml"
        if not os.path.exists(hparams_file):
            print(f"DEBUG: {hparams_file} does not exist. Continuing with empty 'hparams' for now.")
            hparams = {}
        else:
            shutil.copy(hparams_file, f"{logdir}/hparams_bak.yaml")
            
            tries = 5
            success = False
            
            for i in range(tries):
                # "with open" logic is new
                with open(hparams_file, "r") as f:
                    hparams = yaml.safe_load(f)
                if hparams is None:
                    print(f"DEBUG: {hparams_file} is 'None'. Trying again.")
                elif hparams == {}:
                    print(f"DEBUG: {hparams_file} is empty. Trying again.")
                elif not isinstance(hparams, dict):
                    print(f"DEBUG: {hparams_file} is not a dictionary but {type(hparams)}. Trying again.")        
                else:
                    success = True
                    break
                time.sleep(1)  
                
            if not success:
                print(f"DEBUG: {hparams_file} is still 'None' after {tries} attempts. Continuing with empty 'hparams' for now.")
                hparams = {}
                    
        hparams[hparam] = value
        with open(hparams_file, "w") as f:
            yaml.dump(hparams, f)

This version of update_hparam is implemented in the branch 27-sporadic-error-when-updating-hparam.

With the above logic we should eventually be able to catch the error and then I'll implement a clean fix.

Again thanks your help!

Best,
Simon

[zijing-zhang97]

Hi simon，
Thank you for developing this software.
I'm encountering the same error:
TypeError: 'NoneType' object does not support item assignment

Some .mrc files work, while others fail.

When the error occurs, the hparams.yaml contains:

adam_params:
  lr: 0.0002
subtomo_dir: .//subtomos
unet_params:
  chans: 32
  drop_prob: 0.2
  normalization_loc: 12.104392051696777
  normalization_scale: 3.3607137203216553
  num_downsample_layers: 3
update_subtomo_missing_wedges_every_n_epochs: 20

When it works correctly, the hparams.yaml contains:

adam_params:
  lr: 0.0004
subtomo_dir: .//subtomos
unet_params:
  chans: 32
  drop_prob: 0.0
  num_downsample_layers: 3
update_subtomo_missing_wedges_every_n_epochs: 20

---

Code Configuration

shared:
  project_dir: "./"
  tomo0_files: 
    - "./combined_EVN_bin4_rec.mrc"
  tomo1_files: 
    - "./combined_ODD_bin4_rec.mrc"
  subtomo_size: 96
  mw_angle: 50
  num_workers: 64
  gpu: [0,1,2,3]
  seed: 42

prepare_data:
  mask_files:
    - "./combined_bin4_rec_mask.mrc"
  extract_larger_subtomos_for_rotating: true 
  val_fraction: 0.2
  subtomo_extraction_strides: [64, 64, 64]
  overwrite: true

fit_model:
  unet_params_dict:
    chans: 32
    num_downsample_layers: 3
    drop_prob: 0.0
  adam_params_dict: 
    lr: 0.0004
  num_epochs: 1000
  batch_size: 8
  update_subtomo_missing_wedges_every_n_epochs: 20
  check_val_every_n_epochs: 20
  save_n_models_with_lowest_val_loss: 5
  save_n_models_with_lowest_fitting_loss: 5
  save_model_every_n_epochs: 50
  logger: "csv" 
  overwrite: true

---

Error Log

Updating subtomo missing wedges: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 36/36 [00:31<00:00,  1.15it/s]
Computing model-input normalization statistics: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 43.25it/s]
Computing model-input normalization statistics: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 43.62it/s]
Computing model-input normalization statistics: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 42.63it/s]
[rank2]: Traceback (most recent call last):

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
[rank2]:     return trainer_fn(*args, **kwargs)

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 621, in _fit_impl
[rank2]:     self._run(model, ckpt_path=self.ckpt_path)

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1058, in _run
[rank2]:     results = self._run_stage()

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1137, in _run_stage
[rank2]:     self._run_train()

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1160, in _run_train
[rank2]:     self.fit_loop.run()

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
[rank2]:     self.on_advance_end()

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 296, in on_advance_end
[rank2]:     self.trainer._call_lightning_module_hook("on_train_epoch_end")

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1302, in _call_lightning_module_hook
[rank2]:     output = fn(*args, **kwargs)

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/ddw/utils/unet.py", line 85, in on_train_epoch_end
[rank2]:     self.update_normalization()

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/ddw/utils/unet.py", line 166, in update_normalization
[rank2]:     self.update_hparam("unet_params", self.unet_params)

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/ddw/utils/unet.py", line 178, in update_hparam
[rank2]:     hparams[hparam] = value

[rank2]: TypeError: 'NoneType' object does not support item assignment


[rank2]: During handling of the above exception, another exception occurred:


[rank2]: Traceback (most recent call last):

[rank2]:   File "/home/zirui/.conda/envs/ddw/bin/ddw", line 8, in <module>
[rank2]:     sys.exit(main())

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/ddw/app.py", line 15, in main
[rank2]:     app()

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/ddw/fit_model.py", line 273, in fit_model
[rank2]:     trainer.fit(

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 579, in fit
[rank2]:     call._call_and_handle_interrupt(

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 59, in _call_and_handle_interrupt
[rank2]:     trainer.strategy.reconciliate_processes(traceback.format_exc())

[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 461, in reconciliate_processes
[rank2]:     raise DeadlockDetectedException(f"DeadLock detected from rank: {self.global_rank} \n {trace}")

[rank2]: pytorch_lightning.utilities.exceptions.DeadlockDetectedException: DeadLock detected from rank: 2
[rank2]:  Traceback (most recent call last):
[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
[rank2]:     return trainer_fn(*args, **kwargs)
[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 621, in _fit_impl
[rank2]:     self._run(model, ckpt_path=self.ckpt_path)
[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1058, in _run
[rank2]:     results = self._run_stage()
[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1137, in _run_stage
[rank2]:     self._run_train()
[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1160, in _run_train
[rank2]:     self.fit_loop.run()
[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
[rank2]:     self.on_advance_end()
[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 296, in on_advance_end
[rank2]:     self.trainer._call_lightning_module_hook("on_train_epoch_end")
[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1302, in _call_lightning_module_hook
[rank2]:     output = fn(*args, **kwargs)
[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/ddw/utils/unet.py", line 85, in on_train_epoch_end
[rank2]:     self.update_normalization()
[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/ddw/utils/unet.py", line 166, in update_normalization
[rank2]:     self.update_hparam("unet_params", self.unet_params)
[rank2]:   File "/home/zirui/.conda/envs/ddw/lib/python3.10/site-packages/ddw/utils/unet.py", line 178, in update_hparam
[rank2]:     hparams[hparam] = value
[rank2]: TypeError: 'NoneType' object does not support item assignmen。

---

If you could offer any help, I would really appreciate it. Thank you so much!

[Phaips]

Hi @zijing-zhang97 do you run it with SLURM?

[zijing-zhang97]

Hi @zijing-zhang97 do you run it with SLURM?

I don't run it with SLURM

[SimWdm]

@zijing-zhang97 could you try re-running model fitting on the new branch I created for this issue as described here #27 (comment)?

This will probably not fix the issue but will hopefully give us a better error trace! 🙏

Sorry that I cannot do this myself, as I have never encountered this issue in our servers 😕

[SimWdm]

Another thing that has just come to my mind: Has the error ever occured when using a single GPU for model fitting?