Hickle all defined variables

[mpanighel]

As suggested in #28 it would be really useful to dump all the currently defined variables. Optionally they could all go into a single, well structured .hkl file.

As reference here there is an example of user defined variable retrival.

[mpanighel]

Simple proof of concept (needs IPython/Jupyter):

import hickle as hkl
import numpy as np

hkl.dump(0, 'test.hkl', path="dummy", mode='w')#just to have a file to append to

# Define some variables
a = 1
b = "Hello"
array_obj = np.ones(32768, dtype='float32')

#List variables (types should be extended e.g. to pandas DataFrame..)
var_list = %who_ls int float ndarray str
var_dict = {}
for var in var_list:
    var_dict[var] = eval(var)

#Dump defined variables
for k, v in var_dict.items():
    hkl.dump(v, 'test.hkl', path=k, mode='a')

[1313e]

I have looked into this a few times over the past 6 months, but there are many, many different pitholes and other problems to make this work consistently on all three operating systems.
The main issue is to figure out what exactly should be saved.
For example, in an IPython console, there are many variables that the console itself uses for various things, which should obviously not be saved.

[mpanighel]

I have looked into this a few times over the past 6 months, but there are many, many different pitholes and other problems to make this work consistently on all three operating systems.

Ok, I was not aware of these differences in IPython implementations.

The main issue is to figure out what exactly should be saved.
For example, in an IPython console, there are many variables that the console itself uses for various things, which should obviously not be saved.

%who_ls by default keeps track and returns a list of just user-defined variables, as opposed to dir(). So any builtin and module-related variables are disregarded and the returned variables should be all the ones to be saved.

[1313e]

The %who_ls command is a magic command, that works inside an IPython console.
However, it cannot be used by any other script, so I cannot use it to figure out what the local variables are.

[hernot]

Unless you want to use inspect.stack and walk call stack up by one level at least i do fear it is not really doable in a portable manner.
I have done similar thing in an other project to implement sort of public read only __slots__ of classes inspecting stack while applying some Meta-Class magic but it is quite tricky to get things proper.

The easiest would be to call hickle with output of locals() command as object to dump. The down is that this is identical to output of globals() if outside any function or class. This thus does also include all references to imported modules etc and therefore would require specific filtering.

Therefore i do think %who_ls is some magic command specific to IPython kernel which directly interprets this literal and returns the list of variable definitions recorded before.

EDIT
The closest i can think of if lockin to ipython is ok is the following:

def dump_defined_variables(names,localvars,file):
     hickle.dump({name:localvars.get(name,None) for name in names },file)

currently_defined = %who_ls

dump_defined_variables(currentls_defined,locals(),"current_state.h5")

And not more is possible as %who_ls is definitely some ipython kernel special tag. On ipython console i can call it with or without % and it is possible to directly assign as in above example to a local variable but it is not possible to directly pass its output to a function. Writing dump_defined_variables(%who_ls,locals(),"current_state.h5") causes a syntax error complaining about the % sign removing it causes undefined name for who_ls and calling it as function returns empty output. Thus the above is the closest possible on ipython using who_ls and mocking it will be quite tricky if possible at all. Besides %who_ls does not just list variable definitions but also all imports and function definitions and class definitions thus i do think this is just an ipython alias without any special magic for locals().keys() command filtering any item with a key starting with _. In other words define a module private variable like _a=12 and it will not appear in the output of %who_ls. Admitted that case is quite awkward and strange for interactive imperative programming but in conjunction with listing all definitions not just variables it provides some indications upon inner workings of %who_ls ipython special command token. But dumping function definitions is too, especially as even pickle does not store function and class code just its qualified name and the module it was defined by. Thus these local definitions are lost anyway.

Given this the above example could be changed as follows:

hickle.dump({name:value for name,value in locals().items() if name[0] != '_' },"current_state.h5")

# restore using
locals().update(hickle.load("current_state.h5))