Persistent cache with Zarr

[alimanfoo]

This is somewhat related to both #5 and #3 but slightly different. Basically I would like a persistent cache for use when working in a jupyter notebook. The motivation is very similar to https://github.com/rossant/ipycache i.e. if I restart a notebook I don't want to have to repeat any computations that previously finished. However I would like to use Zarr to store cache results not pickle because compression will save disk space. Also I would like to use a memoize function decorator rather than a cell magic, i.e., something more like the cachey memoize decorator and the joblib Memory.cache decorator.

No problem if this is beyond scope for cachey but I thought I'd mention it in case there were any synergies with other requirements. On the technical side there are two main points to consider: one is how to generate a key from function arguments that is stable across python sessions (i.e., doesn't rely on Python's built-in hash function); the second is how to integrate with Zarr (or similar) for storage.

[mrocklin]

Zict may also be useful here

[mrocklin]

It seems like the outputs of a jupyter notebook would be more general than Zarr generally consumes. Is this correct?

[alimanfoo]

Yes, Zarr currently could only cache arrays, although you could hack around to also store scalar values.

FWIW I just hacked something up for my own use, code here: https://gist.github.com/alimanfoo/ed724d207d859ac507696c3f6735fdf9

[mrocklin]

Any thoughts on http://zict.readthedocs.io/en/latest/

[alimanfoo]

The composability of zict is very nice. But for my use case I am mostly wanting to cache results which are numpy arrays, and so I wondered about using Zarr because serialization/deserialization of arrays is very efficient. Also zict doesn't provide a memoize decorator, although it could be adapted to do so. Maybe talking at cross-purposes though, not sure where you were thinking zict to fit in.

…

On Tuesday, January 17, 2017, Matthew Rocklin ***@***.***> wrote: Any thoughts on http://zict.readthedocs.io/en/latest/ — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8Qjh6jYxq3tR6Uk7QuD57Mmf3Es4rks5rTOclgaJpZM4LlbmQ> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: alimanfoo@googlemail.com Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

[mrocklin]

Generally I think the way to go about this is to let cachey consume a MutableMapping rather than construct its own dict. Then all of the choices of compression/serialization etc. fall on the user. Zict composes nicely here because it could be used to construct MutableMappings, even from zarr functions.

One could do this now.

from cachey import Cache
c = Cache(...)
c.data = my_mutable_mapping

[alimanfoo]

OK, that makes sense. The other issue is about making persistent keys, i.e., keys that are stable across different Python sessions. In particular, for the memoize decorator, this means making a stable hash of positional and keyword arguments. Would you see this as in scope for cachey? In joblib.Memory the general approach is to pickle arguments then use hashlib (i.e., cryptographic hashing), although there's lots of extra logic for handling arrays, sets, and trying to normalize positional and keyword args, which I think is a bit over-complicated. In the zarr_cache.py gist I just linearized positional and keyword arguments then pickled and passed through md5, but designed so other methods of producing keys could be plugged in.

…

On Tuesday, January 17, 2017, Matthew Rocklin ***@***.***> wrote: Generally I think the way to go about this is to let cachey consume a MutableMapping rather than construct its own dict. Then all of the choices of compression/serialization etc. fall on the user. Zict composes nicely here because it could be used to construct MutableMappings, even from zarr functions. One could do this now. from cachey import Cache c = Cache(...) c.data = my_mutable_mapping — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8Qrpixb_63xbbBdnL-FXG1BMG9SI7ks5rTQRVgaJpZM4LlbmQ> .

[mrocklin]

Is it possible to keep key generation within the choice of MutableMapping? At the moment cachey consumes python objects for keys (args, kwargs). Presumably it asks the MutableMapping for the value at a key like this and the MutableMapping transforms those python objects into the sort of key that it needs in order to access its internal data structure. If its internal data structure is just a dict then this is trivial. If it is an on-disk file then it has to be more complex as you suggest. At the end of the day though, I think that cachey might be able to say "I don't care. It's up to the MutableMapping how it wants to handle this."

[alimanfoo]

Yes I guess much of this could be pushed behind the MutableMapping interface (it hides a multitude of sins :-). The only thing then to standardize on the cachey side is how to form a key from args and kwargs. Did you mean to just use a tuple of (args, kwargs)?

…

On Wednesday, January 18, 2017, Matthew Rocklin ***@***.***> wrote: Is it possible to keep key generation within the choice of MutableMapping? At the moment cachey consumes python objects for keys (args, kwargs). Presumably it asks the MutableMapping for the value at a key like this and the MutableMapping transforms those python objects into the sort of key that it needs in order to access its internal data structure. If its internal data structure is just a dict then this is trivial. If it is an on-disk file then it has to be more complex as you suggest. At the end of the day though, I think that cachey might be able to say "I don't care. It's up to the MutableMapping how it wants to handle this." — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QlFYUbisSi7TjAJMQ1Joif-DM9_9ks5rThKLgaJpZM4LlbmQ> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: alimanfoo@googlemail.com Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

[mrocklin]

Yeah, it looks like currently the memoize decorator takes in a key function and that key function takes in args and kwargs. Here is the default:

def memo_key(args, kwargs):
    result = (args, frozenset(kwargs.items()))
    try:
        hash(result)
    except TypeError:
        result = tuple(map(id, args)), str(kwargs)
    return result

This is not exactly the best possible answer. Using dask.base.tokenize here might be a good idea:

In [1]: from dask.base import tokenize

In [2]: tokenize([1, 2, 3], {1: 2, 3: {4}})
Out[2]: 'a46c1482d53199982e0211802af8dee6'

In [3]: tokenize([1, 2, 3], {1: 2, 3: {4}})
Out[3]: 'a46c1482d53199982e0211802af8dee6'

[alimanfoo]

The tokenize function from dask looks like a nice solution to the key generation problem. One question, how does a token get generated for an arbitrary Python object? Couldn't make sense of this: @normalize_token.register(object) def normalize_object(o): if callable(o): return normalize_function(o) else: return uuid.uuid4().hex

…

On Tuesday, January 24, 2017, Matthew Rocklin ***@***.***> wrote: Yeah, it looks like currently the memoize decorator takes in a key function and that key function takes in args and kwargs. Here is the default: def memo_key(args, kwargs): result = (args, frozenset(kwargs.items())) try: hash(result) except TypeError: result = tuple(map(id, args)), str(kwargs) return result This is not exactly the best possible answer. Using dask.base.tokenize here might be a good idea: In [1]: from dask.base import tokenize In [2]: tokenize([1, 2, 3], {1: 2, 3: {4}}) Out[2]: 'a46c1482d53199982e0211802af8dee6' In [3]: tokenize([1, 2, 3], {1: 2, 3: {4}}) Out[3]: 'a46c1482d53199982e0211802af8dee6' — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QvhNRIJfmaATNf0URSJtrWj_avTwks5rVkWPgaJpZM4LlbmQ> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: alimanfoo@googlemail.com Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

[mrocklin]

We serialize functions with cloudpickle and then hash them down. For objects that we don't know how to hash we generate a random string. In the context of caching this doesn't make sense and we would just want to not cache.

[alimanfoo]

Would it not be an option to fall back to pickling objects then hashing the pickle string?

…

On Wednesday, January 25, 2017, Matthew Rocklin ***@***.***> wrote: We serialize functions with cloudpickle and then hash them down. For objects that we don't know how to hash we generate a random string. In the context of caching this doesn't make sense and we would just want to not cache. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8Qh4tDOGZsOOLVAl5tuuD6D7YXfBoks5rV0QbgaJpZM4LlbmQ> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: alimanfoo@googlemail.com Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

[mrocklin]

That would certainly be an option. We probably won't do it in Dask for speed and robustness reasons. We're pretty comfortable with false negatives.

[alimanfoo]

OK, so a possible architecture...

Cachey provides a Cache class.

You can set any MutableMapping as the value of the data property on an instance of Cache.

A Cache instance provides a memoize decorator, which uses (args, kwargs) as the key when setting and getting items from the cache. It's up to the data mapping how to transform (args, kwargs) into a key used internally.

To get caching which works across Python sessions, implement a zict-style composable MutableMapping class (call it Hash for now) which does the job of turning (args, kwargs) key into some hashed key and then passes through get/set operations to an inner MutableMapping which handles storage.

To get persistent storage of cached values, implement a storage MutableMapping class (call it Store for now) which uses whatever strategy to save values to disk. This could be using Zarr only save any array-like values and barf on any non-array-like values, or it could use Zarr to save array-like values and some other strategy to save other types of values. Point is, details of persistence are hidden behind this layer.

So then you could do something like:

cache = cachey.Cache()
cache.data = Hash(Store('/path/to/cache'))

@cache.memoize
def myfunc(...):
   ...

The Hash and Store mappings could be combined into a single class, but then you would lose flexibility to combine different hashing and storage implementations.

To generalize, there is a three-layer Cache/Hash/Store architecture. The top layer Cache class could be cachey.Cache or zict.LRU or something simple that caches everying. I'm assuming the top layer Cache class also provides the memoize decorator, or maybe that could be decoupled, but needs to live somewhere.

Is this sounding like something worth pursuing? In Cachey? In Zict?

What I need for my own use case is a simple, flexible way to memoize a function that efficiently stores array-like and scalar return values and works across Python sessions. The additional logic of choosing when to evict cached values based on computational cost is actually a bit unnecessary when storing cached values on disk, although maybe not always if amount of stored data is large.