Fix the hanging issue in merge_piecewise_dataset#236
Conversation
|
📄 Preview your docs here: |
Codecov Report✅ All modified and coverable lines are covered by tests. 🚀 New features to boost your workflow:
|
| concat_dim="time", | ||
| coords="minimal", | ||
| parallel=False, | ||
| lock=threading.Lock(), |
There was a problem hiding this comment.
Could you describe why this lock is needed. Also, have you checked if this work with dask? A test or two would be nice.
There was a problem hiding this comment.
Claude suggested, but basically sometimes the open_mfdataset hangs (it's hard to reproduce why) so know I added a couple args Claude suggested to make hanging less likely. It's pretty clear it's a threading thing, so hopeful this Lock (which means only worker can read at a time) will stop those hanging issues. I've also added a test that on ocassion hangs in a similar places when we don't have this change.
manishvenu
changed the title
2 participants
Try fixing merge_piecewise threading issue