can Nexradlevel2 reader read streamed data?

[aladinor]

Tracked also in #258 and #264

Xradar should support compress files as well as data streaming

pyart library allows to open in different formats including Bz, gz and files stored in an s3 bucket (data) streaming. I think we might want to implement a similar solution to xradar reader.

• Create/adapt prepare_for_read
• Adapt xradar backends to support data streaming as in PyArt

[kmuehlbauer]

@aladinor Thanks for splitting this out. Thanks @ghiggi for raising this.

I'd strongly vote against doing something like this. Instead we should advertise users how to do this with boilerplate code.

Things work for PyArt because they import the whole file into memory. This is something we avoid with the current implementation trying to only read in necessary data.

[aladinor]

Hi @kmuehlbauer,

Thanks for pointing that out. I totally agree that we should avoid loading the full file into memory and rely on numpy.memmap to access only the necessary data. However, I’m curious—how large do these files typically get for this to be a significant issue?

It seems like using numpy.memmap is a great approach for local files, but when dealing with remote data (e.g., NEXRAD files stored on AWS S3), it requires downloading the entire file first. Streaming might be a useful alternative here, even if it comes with some trade-offs. For instance, the Sigmet backend supports streaming, and it works well in similar cases.

Since most NEXRAD data is stored in S3, it might be worth considering an optional streaming implementation.

Please let me know your thoughts

[kmuehlbauer]

So "support streaming" should just mean "can read streamed data"? Then we might just adapt this feature from IRIS to NEXRAD to make this happen.

What I'm voting against is to use gz/s3fs/fsspec/etc inside our implementation.

[aladinor]

Hi @kmuehlbauer,

Thanks for the clarification, and my apologies for the confusion—that was entirely my misunderstanding! You're absolutely right. By "support streaming," we’re aiming for the ability to read streamed data, not necessarily to integrate gz, s3fs, or fsspec directly into the core implementation.

Given that, adapting the streaming capabilities from the IRIS reader for NEXRAD sounds like a great idea. I’ll take a closer look at how we can make that happen without adding unnecessary dependencies.

Thanks again for your patience and guidance on this!

[kmuehlbauer]

@aladinor Great! These additions to the first IRIS reader have been made when it still was a part of wradlib. I can look up the commits and link them here, if necessary.

[aladinor]

Hi @kmuehlbauer,

That sounds awesome! I'd be happy to work on adapting the IRIS streaming feature for NEXRAD. If you could link those commits, it would definitely be helpful as a reference!

[kmuehlbauer]

@aladinor These are the relevant wradlib PR's:

• First implementation: ENH: Iris/Sigmet importer #33
• Implement np.memmap: ENH: IRIS reader wradlib/wradlib#170
• Redesign (read ingest files): ENH: Sigmet redesign wradlib/wradlib#303
• File-like objects: ENH: make wradlib.io capable of consuming file-like objects wradlib/wradlib#469
• First xarray backend design: ENH: add iris/sigmet xarray backend wradlib/wradlib#520

[ghiggi]

Hi guys !

I just wanted to mention that we could eventually adding the capability to directly read compressed formats and open files from S3 paths into the radar_api wrappers.

Right now, when the radar_api detects an S3 path, it uses fsspec’s simplecache to temporarily download the file to disk and then opens it as if it were a local file.
I have noticed that if there are too many chunks or too many simultaneous connections to S3, the latency involved in connecting to the bucket can make reading or streaming even a small portion of the file slower than if we were to download the entire file first and then open it from local disk.
So keep in mind the amount of overhead caused by S3 connection latency in your design choices.

[kmuehlbauer]

Thanks @ghiggi for letting us know. That behaviour is another indication that xradar should not put too much magic into file readers, but let the user decide how to claim the data. xradar should just be capable to consume any of these files/streams.

[ghiggi]

I just discovered now about obstore - a fsspec reimplemented in rust that seems to allow much faster data streaming. They are also merging the ObStore into Zarr (zarr-developers/zarr-python#1661). Might be worth to support also this type of file objects.

(Maybe @aladinor might want to also check the speed improvement for IRIS/Segmet files 😄)