@@ -15,6 +15,96 @@ To install: ```pip install dol```
1515
1616[ Documentation here] ( https://i2mint.github.io/dol/ )
1717
18+ ## Example use
19+
20+ Say you have a source backend that has pickles of some lists-of-lists-of-strings,
21+ using the ` .pkl ` extension, and you want to copy this data to a target backend,
22+ but saving them as gzipped csvs with the ` csv.gz ` extension.
23+
24+ We'll first work with dictionaries instead of files here, so we can test more easily,
25+ and safely.
26+
27+ ``` python
28+ import pickle
29+
30+ src_backend = {
31+ ' file_1.pkl' : pickle.dumps([[' A' , ' B' , ' C' ], [' one' , ' two' , ' three' ]]),
32+ ' file_2.pkl' : pickle.dumps([[' apple' , ' pie' ], [' one' , ' two' ], [' hot' , ' cold' ]]),
33+ }
34+ targ_backend = dict ()
35+ ```
36+
37+ Here's how you can do it with ` dol ` tools
38+
39+ ``` python
40+ from dol import ValueCodecs, KeyCodecs, Pipe
41+
42+ # decoder here will unpickle data and remove remove the .pkl extension from the key
43+ src_wrap = Pipe(KeyCodecs.suffixed(' .pkl' ), ValueCodecs.pickle())
44+
45+ # encoder here will convert the lists to csv string, the string into bytes,
46+ # and the bytes will be gzipped.
47+ # ... also, we'll add .csv.gz on write.
48+ targ_wrap = Pipe(
49+ KeyCodecs.suffixed(' .csv.gz' ),
50+ ValueCodecs.csv() + ValueCodecs.str_to_bytes() + ValueCodecs.gzip()
51+ )
52+
53+ # Let's wrap our backends:
54+ src = src_wrap(src_backend)
55+ targ = targ_wrap(targ_backend)
56+
57+ # and copy src over to targ
58+ print (f " Before: { list (targ_backend)= } " )
59+ targ.update(src)
60+ print (f " After: { list (targ_backend)= } " )
61+ ```
62+
63+ From the point of view of src and targ, you see the same thing.
64+
65+ ``` python
66+ assert list (src) == list (targ) == [' file_1' , ' file_2' ]
67+ assert (
68+ src[' file_1' ]
69+ == targ[' file_1' ]
70+ == [[' A' , ' B' , ' C' ], [' one' , ' two' , ' three' ]]
71+ )
72+ ```
73+
74+ But the backend of targ is different:
75+
76+ ``` python
77+ src_backend[' file_1.pkl' ]
78+ # b'\x80\x04\x95\x19\x00\x00\x00\x00\x00\x00\x00]\x94(]\x94(K\x01K\x02K\x03e]\x94(K\x04K\x05K\x06ee.'
79+ targ_backend[' file_1.csv.gz' ]
80+ # b'\x1f\x8b\x08\x00*YWe\x02\xff3\xd41\xd21\xe6\xe52\xd11\xd51\xe3\xe5\x02\x00)4\x83\x83\x0e\x00\x00\x00'
81+ ```
82+
83+ Now that you've tested your setup with dictionaries, you're ready to move on to real,
84+ persisted storage. If you wanted to do this with local files, you'd:
85+
86+ ``` python
87+ from dol import Files
88+ src = Files(' PATH_TO_LOCAL_SOURCE_FOLDER' )
89+ targ = Files(' PATH_TO_LOCAL_TARGET_FOLDER)
90+ ```
91+
92+ But you could do this with AWS S3 using tools from
93+ [s3dol](https:// github.com/ i2mint/ s3dol), or Azure using tools from
94+ [azuredol](https:// github.com/ i2mint/ azuredol), or mongoDB with
95+ [mongodol](https:// github.com/ i2mint/ mongodol),
96+ github with [hubcap](https:// github.com/ thorwhalen/ hubcap), and so on...
97+
98+ All of these extensions provide adapters from various data sources/ targets to the
99+ dict - like interface (called " Mapping" in python typing).
100+ What `dol` provides are base tools to make a path from these to the interface
101+ that makes sense for the domain, or business logic in front of you,
102+ so that you can purify your code from implementation details, and therefore be
103+ create more robust and flexible code as far as data operations are concerned.
104+
105+
106+ # # Historical note
107+
18108Note: This project started as [`py2store` ](https:// github.com/ i2mint/ py2store).
19109`dol` is the core of py2store has now been factored out
20110and many of the specialized data object layers moved to separate packages.
0 commit comments