Comparison to `scverse/shadows`

[mschilli87]

I there any comparison (feature and performance-wise) of this module and scverse/shadows? To me it seems like they serve very similar (if not the same) purpose but with a different API (eg. implicit vs. explicit caching/loading).

I am asking here as the other module existed for almost a year when this one started so I am assuming there must have been a use case not supported by shadows that motivated the development of cap-anndata.

I am asking as I find it hard to chose which of the two modules to commit to for partial/lazy loading of AnnData objects from H5ad files for my pipeline(s).

[rm1113]

Hi @mschilli87, thanks for the interest!
We didn't compare our package with scverse/shadows. The package was created especially for Cell Annotation Platform. The only thing I can say for sure, we have used this package in prod for a long time and have no complaints :)

Our main use case is to read and update in-place obs/var/uns section. If you have any comparison, will be happy to discuss

[rm1113]

Hi, @mschilli87
Out of curiosity, what did you decide and why?

[mschilli87]

So far I have only used scverse/shadows. The main reason to start with that one was that I consider it more 'official' by being hosted under the scverse organisation. I.e. the people behind that package are directly involved in the development of the AnnData format and related packages. Therefore, I expect equal if not better API coverage and faster adaption to future changes in the long run.
However, judging by the current state scverse/shadows lacks a few features I had to work around (some related to handling the .raw element, which I expect to be deprecated in a future version of the AnnData specs though; another one would be the lack of write_* functions).

If you have time for a proper comparison, please let me know.
Else, I hope to eventually find the time to checkout this alternative myself.

[rm1113]

one would be the lack of write_* functions).

Initially, the whole point of cap-anndata was to implement those functions. We didn't want to read a whole AnnData object to update a single uns field or rename one of the categories in the obs column. So, I think it is the main difference between our package and shadows atm.

[mschilli87]

I think they do support updating a file without loading all data. My use case was to lazy-load an H5AD file, subset the data, and write the result to a new H5AD file. That last step is what I couldn't do directly, as their shadow classes don't provide a write_h5ad function (yet).