@@ -15,6 +15,96 @@ To install: ```pip install dol```
15
15
16
16
[ Documentation here] ( https://i2mint.github.io/dol/ )
17
17
18
+ ## Example use
19
+
20
+ Say you have a source backend that has pickles of some lists-of-lists-of-strings,
21
+ using the ` .pkl ` extension, and you want to copy this data to a target backend,
22
+ but saving them as gzipped csvs with the ` csv.gz ` extension.
23
+
24
+ We'll first work with dictionaries instead of files here, so we can test more easily,
25
+ and safely.
26
+
27
+ ``` python
28
+ import pickle
29
+
30
+ src_backend = {
31
+ ' file_1.pkl' : pickle.dumps([[' A' , ' B' , ' C' ], [' one' , ' two' , ' three' ]]),
32
+ ' file_2.pkl' : pickle.dumps([[' apple' , ' pie' ], [' one' , ' two' ], [' hot' , ' cold' ]]),
33
+ }
34
+ targ_backend = dict ()
35
+ ```
36
+
37
+ Here's how you can do it with ` dol ` tools
38
+
39
+ ``` python
40
+ from dol import ValueCodecs, KeyCodecs, Pipe
41
+
42
+ # decoder here will unpickle data and remove remove the .pkl extension from the key
43
+ src_wrap = Pipe(KeyCodecs.suffixed(' .pkl' ), ValueCodecs.pickle())
44
+
45
+ # encoder here will convert the lists to csv string, the string into bytes,
46
+ # and the bytes will be gzipped.
47
+ # ... also, we'll add .csv.gz on write.
48
+ targ_wrap = Pipe(
49
+ KeyCodecs.suffixed(' .csv.gz' ),
50
+ ValueCodecs.csv() + ValueCodecs.str_to_bytes() + ValueCodecs.gzip()
51
+ )
52
+
53
+ # Let's wrap our backends:
54
+ src = src_wrap(src_backend)
55
+ targ = targ_wrap(targ_backend)
56
+
57
+ # and copy src over to targ
58
+ print (f " Before: { list (targ_backend)= } " )
59
+ targ.update(src)
60
+ print (f " After: { list (targ_backend)= } " )
61
+ ```
62
+
63
+ From the point of view of src and targ, you see the same thing.
64
+
65
+ ``` python
66
+ assert list (src) == list (targ) == [' file_1' , ' file_2' ]
67
+ assert (
68
+ src[' file_1' ]
69
+ == targ[' file_1' ]
70
+ == [[' A' , ' B' , ' C' ], [' one' , ' two' , ' three' ]]
71
+ )
72
+ ```
73
+
74
+ But the backend of targ is different:
75
+
76
+ ``` python
77
+ src_backend[' file_1.pkl' ]
78
+ # b'\x80\x04\x95\x19\x00\x00\x00\x00\x00\x00\x00]\x94(]\x94(K\x01K\x02K\x03e]\x94(K\x04K\x05K\x06ee.'
79
+ targ_backend[' file_1.csv.gz' ]
80
+ # b'\x1f\x8b\x08\x00*YWe\x02\xff3\xd41\xd21\xe6\xe52\xd11\xd51\xe3\xe5\x02\x00)4\x83\x83\x0e\x00\x00\x00'
81
+ ```
82
+
83
+ Now that you've tested your setup with dictionaries, you're ready to move on to real,
84
+ persisted storage. If you wanted to do this with local files, you'd:
85
+
86
+ ``` python
87
+ from dol import Files
88
+ src = Files(' PATH_TO_LOCAL_SOURCE_FOLDER' )
89
+ targ = Files(' PATH_TO_LOCAL_TARGET_FOLDER)
90
+ ```
91
+
92
+ But you could do this with AWS S3 using tools from
93
+ [s3dol](https:// github.com/ i2mint/ s3dol), or Azure using tools from
94
+ [azuredol](https:// github.com/ i2mint/ azuredol), or mongoDB with
95
+ [mongodol](https:// github.com/ i2mint/ mongodol),
96
+ github with [hubcap](https:// github.com/ thorwhalen/ hubcap), and so on...
97
+
98
+ All of these extensions provide adapters from various data sources/ targets to the
99
+ dict - like interface (called " Mapping" in python typing).
100
+ What `dol` provides are base tools to make a path from these to the interface
101
+ that makes sense for the domain, or business logic in front of you,
102
+ so that you can purify your code from implementation details, and therefore be
103
+ create more robust and flexible code as far as data operations are concerned.
104
+
105
+
106
+ # # Historical note
107
+
18
108
Note: This project started as [`py2store` ](https:// github.com/ i2mint/ py2store).
19
109
`dol` is the core of py2store has now been factored out
20
110
and many of the specialized data object layers moved to separate packages.
0 commit comments