Add docs for accessing cogs with terra #125
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
View / edit / reply to this conversation on ReviewNB wildintellect commented on 2025-01-14T00:59:03Z Be clear the NO SIGN is particular to this dataset that's in a public bucket. Should we also add
|
|
View / edit / reply to this conversation on ReviewNB wildintellect commented on 2025-01-14T00:59:04Z this is an interesting approach, nice because it operates with only 1 file, but a little weird at the same time in terms of arbitrarily calculating some neighboring extents based on the image.
I wonder if it's better to have a known bbox and two images to pull partials. I don't really want to complicate it by adding STAC (which we have some example to borrow from)
Maybe this would make sense with a plot of the full extent that has a box from the crop extents to be used. |
|
I agree the 1/100th square is pretty arbitrary. Perhaps it would make sense to build out and draw geojson for the extents to grab and the geotiff tiles to motivate the role of tiling in reducing necessary requests? |
|
I also realized that this example only shows AWS, we probably need a note that this method is valid for all "vsi" methods in gdal, and that the |
|
I included a little note about VSI and a reference to the docs that explain such filesystems. I've also replaced the 1/100th (arbitrary) read with a couple of reads that specifically articulate the impact of internal tile structure on how (and how many) requests get made |
|
So far, there's no |
|
View / edit / reply to this conversation on ReviewNB zacdezgeo commented on 2025-01-15T16:17:26Z maybe we can link the https://guide.cloudnativegeo.org/ in the introduction? |
|
View / edit / reply to this conversation on ReviewNB jsignell commented on 2025-01-15T18:14:17Z I think in the very first sentence you should say what COGs are (a file format for storing raster data with a 2D aligned grid?)
The key benefits section feels like some double counting. I think you should remove "Subset Access" and "Cloud-Native Efficiency" those feel like more the mechanisms that enable "Cost Savings" and "Fast Workflows". Also don't COGs enable pyramiding? That feels like it could be mentioned somehow as well. moradology commented on 2025-01-15T19:55:36Z Good call. I even explicitly use overviews later on so I should probably acknowledge that |
|
View / edit / reply to this conversation on ReviewNB jsignell commented on 2025-01-15T18:14:18Z This is kind of a meta comment, but I tend to glaze over inline comments in notebooks. Maybe just put these each in their own cell with markdown cells describing what is happening.
moradology commented on 2025-01-15T19:54:49Z Wondered about that. They're so nice for getting the information right up next to the place where it has context but I think I do the same... |
|
View / edit / reply to this conversation on ReviewNB jsignell commented on 2025-01-15T18:14:18Z The tile information is block size and layout right? Statistics are block level right? Might be useful to make that explicit.
|
|
View / edit / reply to this conversation on ReviewNB jsignell commented on 2025-01-15T18:14:19Z what are these axes? I guess they are meters in EPSG 9807 or something, but good to make it clear |
|
View / edit / reply to this conversation on ReviewNB jsignell commented on 2025-01-15T18:14:20Z Just a question I had in reading this: can you read the header for the COG without downloading the file?
Also: is there a way to prove that only one tile was downloaded? wildintellect commented on 2025-01-15T21:52:42Z
moradology commented on 2025-01-22T16:17:51Z In the python docs, I have some custom logging that demonstrates the byte-ranges being read. Unfortunately, not so easily done with terra's use of GDAL |
|
View / edit / reply to this conversation on ReviewNB jsignell commented on 2025-01-15T18:14:21Z Line #6. block_size_y <- block_size_pixels * r_res[2] Do you think it would be clearer to just give a bounding box without showing how it is created? If we think about a real world scenario it doesn't matter that people know how to create the box. Actually maybe the whole code cell here should just be hidden since the visual serves a didactic purpose. Same with the other plot that shows gridlines. moradology commented on 2025-01-15T20:06:31Z I think I disagree on this point because it might be good to show how these APIs for pixel and geospatial coordinates work. I'll change things over if your take here is seconded though - I can totally see your point wildintellect commented on 2025-01-16T00:10:50Z the question is who is the target audience? I think the initial target is data users who almost always just need a bbox (or polygon) of data. I'm ok hiding some of that magic from them, maybe a collapsed cell block is the right answer. |
|
Wondered about that. They're so nice for getting the information right up next to the place where it has context but I think I do the same... View entire conversation on ReviewNB |
|
Good call. I even explicitly use overviews later on so I should probably acknowledge that View entire conversation on ReviewNB |
|
I think I disagree on this point because it might be good to show how these APIs for pixel and geospatial coordinates work. I'll change things over if your take here is seconded though - I can totally see your point View entire conversation on ReviewNB |
View entire conversation on ReviewNB |
|
the question is who is the target audience? I think the initial target is data users who almost always just need a bbox (or polygon) of data. I'm ok hiding some of that magic from them, maybe a collapsed cell block is the right answer. View entire conversation on ReviewNB |
|
In the python docs, I have some custom logging that demonstrates the byte-ranges being read. Unfortunately, not so easily done with terra's use of GDAL View entire conversation on ReviewNB |
This PR adds some documentation about accessing data within cogs in R with
terrawithout loading the whole file. Some (as yet unclear) combination of terra and the r kernel's behaviors prevented the kind of byte-range logging in #124, so this short guide sticks to demonstrating the API and describing its cloud-optimized behaviors