Subpath filtering in path-keyed store

[thorwhalen]

A good proportion of our stores have "path keys"...

• ... a good proportion of those are strings
• ... a good proportion of these representing keys into something like a file-like system

What ever the case, subpaths always can represent a subgroup of keys (that start with that subpath), and often we'd like to have something that ressembles "show me only files/subfolders of some folder kind of functionality, like:

# normal behavior:
v = store['some/path/to/a/value'] 
# and here v is the actual object we want, but...
v = store['some/path/'] 
# here v would be a "sub-store" that would only contain values whose keys start with 'some/path/'

Here's some util/helper code to start with (only applies to string paths, but not necessarily ones that represent files and folders).
Note, though, that we can easily generalize to any type of paths by generalizing the "str.startswith" etc.

from dol import wrap_kvs, filt_iter, cached_keys

os_path_sep = os.path.sep

def _prefix_filtering(store, k, v, *, sep=os_path_sep, relative_keys=False, cache_keys=False):
    if k.endswith(sep):
        # TODO: Enable more efficient prefix filtering 
        # (for example, using the store class init param meant for that)
        store = filt_iter(store, filt=lambda key: key.startswith(k))
        if relative_keys:
            raise NotImplementedError("Not implemented relative_keys yet")
        if cache_keys:
            store = cached_keys(store)
        return store
    else:
        return self[k]

Possibly relevant resources

• this discussion about path access

[sylvainbonnot]

In _prefix_filtering, the line store = filt_iter(store, filt=lambda k: k.startswith(k)) should be replaced with store = filt_iter(store, filt=lambda key: key.startswith(k)).

[thorwhalen]

Example of an implementation (well, the tests of the implementation) of this for s3dol:
https://github.com/i2mint/s3dol/blob/migrate-to-s3dol/s3dol/tests/test_base.py#L60-L66