Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ability to read file partially #569

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

ability to read file partially #569

wants to merge 4 commits into from

Conversation

elee1766
Copy link

@elee1766 elee1766 commented Sep 9, 2022

hi!

currently the package can only open a stream from the opened position -> end of file

i store indexed blobs on s3 and perform small random reads on them (static database), so for me, it is useful to open a stream and read only say, 1kb out of a 30gb file

i made this change which makes Read() only request up to the size of the byte buffer passed in. I also added tests related to this, and the current tests pass (at least on my computer)

this does mean that current applications which use readAll on a large file, the request will be broken up into many smaller requests. Whether or not this is preferable, it is not for me to say. The previous behavior can be replicated through passing in a byte buffer with a length larger than or equal to the size of the file, something which is known to the user before they have a chance to call read.

i'm currently using a fork of this library for this purpose, would love to be able to switch back. let me know if there is anything else i would need to do to get this in!

@elee1766
Copy link
Author

hi @fclairamb i saw the activity on this

so i actually stopped work on this branch and wrote a new package combining stuff from this package, another package, and my fork. https://gfx.cafe/open/s3fs/-/blob/master/file.go

more than happy to move a bunch of that work into a new clean fork from your main branch, let me know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants