Error loading of fitting_subtomos

[zijing-zhang97]

Hi simon，

---

Sorry to bother you again. I encountered the following error log while running my program:

Epoch 219: 100%|███████████████████████████████████████████████████████████████████████| 10/10 [00:04<00:00,  2.31it/s, loss=16.9, v_num=1, fitting_loss=17.00, val_loss=16.20]Epoch 219, global step 1760: 'fitting_loss' was not in top 5
Updating subtomo missing wedges:   0%|                                                                                                                  | 0/36 [00:00<?, ?it/s]Error loading .//subtomos/fitting_subtomos/subtomo0/0.pt                                                                                                | 0/36 [00:00<?, ?it/s]
Error message is: Weights only load failed. In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source.
Please file an issue with the following so that we can make `weights_only=True` compatible with your use case: WeightsUnpickler error: Unsupported operand 80

Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html.
Retrying in 1 seconds
Updating subtomo missing wedges: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 36/36 [00:28<00:00,  1.27it/s]

The configuration I used to run the program is:

shared:
  project_dir: "./"
  tomo0_files: 
    - "./combined_EVN_bin4_rec.mrc"
  tomo1_files: 
    - "./combined_ODD_bin4_rec.mrc"
  subtomo_size: 96
  mw_angle: 50
  # standardize_full_tomos: true  # not necessary for this tutorial but very useful if your tomograms have low voxel intensities; see "ddw prepare-data --help" for details
  num_workers: 64
  gpu: [0,1,2,3]  # you can also set this to a list of GPUs, e.g. [0, 1]. Note: Only fit_model will use multiple GPUs. refine_tomogram will use only one GPU (the first one in the list).
  # distributed_backend: "nccl"  # which backend PyTorch Lightning uses for distributed training 
  seed: 42

prepare_data:
  mask_files:
    - "./combined_full_k_bin4_rec_mask.mrc"
  extract_larger_subtomos_for_rotating: true 
  val_fraction: 0.2
  subtomo_extraction_strides: [64, 64, 80]
  standardize-full-tomos: false
  overwrite: true

fit_model:
    unet_params_dict:
      chans: 32
      num_downsample_layers: 3
      drop_prob: 0.0
    adam_params_dict: 
      lr: 0.0004
    num_epochs: 1000
    batch_size: 8
    update_subtomo_missing_wedges_every_n_epochs: 20
    check_val_every_n_epochs: 20
    save_n_models_with_lowest_val_loss: 5
    save_n_models_with_lowest_fitting_loss: 5
    save_model_every_n_epochs: 50
    logger: "csv"
    overwrite: true

I'm not sure whether this error affects the loading of the fitting-subtomos. If you could offer any help, I would really appreciate it. Thank you so much!

[SimWdm]

Hi Zijing,

thanks for trying DeepDeWedge and for reaching out!

The issue looks like it could be related to the torch version.
Could you try downgrading torch to version 2.2.0?
This is the version I used when I developed the code.

Best,
Simon

[zijing-zhang97]

Hi Simon, thank you for your reply. I'll give it a try—really appreciate it!