I/O Bench: automatically download data

[adamjstewart]

When following https://torchgeo.readthedocs.io/en/latest/user/contributing.html#i-o-benchmarking, the config files are supposed to download the data for you (hence checksum: true). At some point, this was removed (probably by me) to avoid our tests downloading data. This PR re-adds automatic downloads (and removes it from Digital Typhoon).

[calebrob6]

When I run python -m torchgeo fit --config tests/conf/io_raw.yaml with this change I get:

Downloading https://cdn-lfs-us-1.hf.co/repos/ee/13/ee13a02ba894b5c9e06fcda4fb167a1a27ba90c0d6bbb4d3c02754c697879f5b/af3fd271b98fd88abbca30bf43b07718b40d32eb40eff0eebf6446da11f43086?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27raw.tar.gz%3B+filename%3D%22raw.tar.gz%22%3B&response-content-type=application%2Fgzip&Expires=1739923706&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTczOTkyMzcwNn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmhmLmNvL3JlcG9zL2VlLzEzL2VlMTNhMDJiYTg5NGI1YzllMDZmY2RhNGZiMTY3YTFhMjdiYTkwYzBkNmJiYjRkM2MwMjc1NGM2OTc4NzlmNWIvYWYzZmQyNzFiOThmZDg4YWJiY2EzMGJmNDNiMDc3MThiNDBkMzJlYjQwZWZmMGVlYmY2NDQ2ZGExMWY0MzA4Nj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSomcmVzcG9uc2UtY29udGVudC10eXBlPSoifV19&Signature=RXHmI7W6aMzrfp2oBc37OSwWtapi1rPM5H0pFadgoiSVD4HjiKIc-j2Ihsk1MOieT6sCs5ZbN4szNaHOIWRyzUC8O39e6Tr6zwPTTuDyo8suLjwhWbdW27I6xuPv3b%7EAg1Qm%7E5eVvjtkkhmyI%7EbTc-OVaFFYFH0ecOqAu73auj2iTBb6JWHJMo1aJ-tkMxioU00wPD4WdVmfv2kKuDXeq64WWWaWiOq0x1OFGjwVTzed3iuX3j2HVM5K6i6pAnHOyBcqFAakrEkwFeYzHJrgxfjxqXpOxaqKvO7LKGBcdudUHwPY5-UHy8cQzRvPoZc4-rilRWVkyZ4eL2ADPNrh9w__&Key-Pair-Id=K24J24Z295AEI9 to data/io/raw.tar.gz
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 546M/546M [00:13<00:00, 41.9MB/s]
Converting CDL CRS from EPSG:5070 to EPSG:32616
Converting CDL res from 30.000255453565156 to 30.0
[rank0]: Traceback (most recent call last):
[rank0]:   File "<frozen runpy>", line 198, in _run_module_as_main                                                                                                                                                                                                                                                                                                                        [rank0]:   File "<frozen runpy>", line 88, in _run_code                                                                                                                                                                                                                                                                                                                                   [rank0]:   File "/home/davrob/ssdprivate/torchgeo/torchgeo/__main__.py", line 10, in <module>
[rank0]:     main()
[rank0]:   File "/home/davrob/ssdprivate/torchgeo/torchgeo/main.py", line 28, in main
[rank0]:     LightningCLI(
[rank0]:   File "/opt/conda/envs/geo/lib/python3.12/site-packages/lightning/pytorch/cli.py", line 396, in __init__
[rank0]:     self._run_subcommand(self.subcommand)
[rank0]:   File "/opt/conda/envs/geo/lib/python3.12/site-packages/lightning/pytorch/cli.py", line 706, in _run_subcommand
[rank0]:     fn(**fn_kwargs)
[rank0]:   File "/opt/conda/envs/geo/lib/python3.12/site-packages/lightning/pytorch/trainer/trainer.py", line 539, in fit
[rank0]:     call._call_and_handle_interrupt(
[rank0]:   File "/opt/conda/envs/geo/lib/python3.12/site-packages/lightning/pytorch/trainer/call.py", line 46, in _call_and_handle_interrupt
[rank0]:     return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/geo/lib/python3.12/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch
[rank0]:     return function(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/geo/lib/python3.12/site-packages/lightning/pytorch/trainer/trainer.py", line 575, in _fit_impl
[rank0]:     self._run(model, ckpt_path=ckpt_path)
[rank0]:   File "/opt/conda/envs/geo/lib/python3.12/site-packages/lightning/pytorch/trainer/trainer.py", line 942, in _run
[rank0]:     self._data_connector.prepare_data()
[rank0]:   File "/opt/conda/envs/geo/lib/python3.12/site-packages/lightning/pytorch/trainer/connectors/data_connector.py", line 94, in prepare_data
[rank0]:     call._call_lightning_datamodule_hook(trainer, "prepare_data")
[rank0]:   File "/opt/conda/envs/geo/lib/python3.12/site-packages/lightning/pytorch/trainer/call.py", line 193, in _call_lightning_datamodule_hook
[rank0]:     return fn(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/davrob/ssdprivate/torchgeo/torchgeo/datamodules/geo.py", line 89, in prepare_data
[rank0]:     self.dataset_class(**self.kwargs)
[rank0]:   File "/home/davrob/ssdprivate/torchgeo/torchgeo/datasets/iobench.py", line 98, in __init__
[rank0]:     super().__init__(self.landsat, self.cdl)
[rank0]:   File "/home/davrob/ssdprivate/torchgeo/torchgeo/datasets/geo.py", line 989, in __init__
[rank0]:     self._merge_dataset_indices()
[rank0]:   File "/home/davrob/ssdprivate/torchgeo/torchgeo/datasets/geo.py", line 1006, in _merge_dataset_indices
[rank0]:     raise RuntimeError('Datasets have no spatiotemporal intersection')
[rank0]: RuntimeError: Datasets have no spatiotemporal intersection

(also it tries running on all GPUs by default)

[adamjstewart]

That is quite strange, it works for me. Also, the GPUs don't get used much, IOBenchTask is a no-op.