Why not implement `HasCursor` for `AsyncRead + AsyncSeek` in `rc-zip-tokio`?

[Diegovsky]

Hey there, I loved using rc-zip-sync for an earlier project. The API is straight to the point and very nice. However, I don't understand how async is done in rc-zip-tokio considering it delegates all file operations to positioned_io. To that end: why not have some wrapper/implement HasCursor for any type that is AsyncRead + AsyncSeek?

[fasterthanlime]

If I recall correctly, the problem is that async read and async seek actually need mutable access to the underlying resource, whereas position to I/O doesn't because it's not changing any state, it's just reading from the file or whatever.

And the whole hasCursor thing is being duplicated when you're reading entries. So the current API is not very well suited to that kind of I/O. And also, just in general, I wanted to avoid doing syscalls to seek around the file when you just have a syscall to read from the right part of the file.

I'm curious what's the underlying IO in your case? Like what are you actually decompressing from? It can't be an HTTP response because those do not implement seek, right?

[Diegovsky]

Nope. I do an http request, stream to a tokio file, seek to the start (though now that I think about it: zip info is located at the end) and then I would love to be able to pass that to rc-zip-tokio so I don't need to reopen the file.

As a workaround, I closed the file and reopened it with RandomAccessFile. It works but it's a bit weird to have tokio in the name and not support a tokio file. Your explanation makes a lot of sense though.

Supposedly it's not ideal because of TTCTOU attacks but I doubt I can't solve this with some other resource locking inside my own app.

[fasterthanlime]

And the reason you're buffering through the disk is because of file size?

Recent versions of RC-zip support decompressing the streaming fashion, so you would not even need seek in that scenario.

Personally, I would recommend doing range requests through HTTP. And for something like that, positioned reads make a lot more sense than seeking and then reading. Or rather, I think it's an easier abstraction to map onto range requests.

But you'd still want to have some block-based buffering strategy to avoid doing too many HTTP requests — you'd want to coalesce multiple small requests into a big request for a block.

I've done that before in Go for Butler, which does installations for the itch.io desktop application. And I meant to do it as a demo for rc-zip as well, but I haven't gotten around to it yet :)

[Diegovsky]

And the reason you're buffering through the disk is because of file size?

Yes, I can't really predict if it would fit in ram.

Recent versions of RC-zip support decompressing the streaming fashion, so you would not even need seek in that scenario.

Isn't this kinda inefficient though? IIRC zip information is stored at the end of the file, so waiting for the whole archive to be in memory, to then start un-archiving feels pretty wrong to me.

Personally, I would recommend doing range requests through HTTP. And for something like that, positioned reads make a lot more sense than seeking and then reading. Or rather, I think it's an easier abstraction to map onto range requests.

But you'd still want to have some block-based buffering strategy to avoid doing too many HTTP requests — you'd want to coalesce multiple small requests into a big request for a block.

I've done that before in Go for Butler, which does installations for the itch.io desktop application. And I meant to do it as a demo for rc-zip as well, but I haven't gotten around to it yet :)

Well, I don't think I can implement such a beast myself! I'm pretty sure this is by far the most correct way, but for the "script but rust" approach I'm going for, it's pretty overkill and I can't guarantee the server (I don't control it) implements range request support.

[fasterthanlime]

Isn't this kinda inefficient though? IIRC zip information is stored at the end of the file, so waiting for the whole archive to be in memory, to then start un-archiving feels pretty wrong to me.

No, it's unsafe, it's using only the local headers instead of using the end of central directory x)