Skip to content

Commit d9e91a1

Browse files
committed
doc: Updated readme
1 parent d6f147f commit d9e91a1

File tree

2 files changed

+91
-1
lines changed

2 files changed

+91
-1
lines changed

README.md

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,96 @@ To install: ```pip install dol```
1515

1616
[Documentation here](https://i2mint.github.io/dol/)
1717

18+
## Example use
19+
20+
Say you have a source backend that has pickles of some lists-of-lists-of-strings,
21+
using the `.pkl` extension, and you want to copy this data to a target backend,
22+
but saving them as gzipped csvs with the `csv.gz` extension.
23+
24+
We'll first work with dictionaries instead of files here, so we can test more easily,
25+
and safely.
26+
27+
```python
28+
import pickle
29+
30+
src_backend = {
31+
'file_1.pkl': pickle.dumps([['A', 'B', 'C'], ['one', 'two', 'three']]),
32+
'file_2.pkl': pickle.dumps([['apple', 'pie'], ['one', 'two'], ['hot', 'cold']]),
33+
}
34+
targ_backend = dict()
35+
```
36+
37+
Here's how you can do it with `dol` tools
38+
39+
```python
40+
from dol import ValueCodecs, KeyCodecs, Pipe
41+
42+
# decoder here will unpickle data and remove remove the .pkl extension from the key
43+
src_wrap = Pipe(KeyCodecs.suffixed('.pkl'), ValueCodecs.pickle())
44+
45+
# encoder here will convert the lists to csv string, the string into bytes,
46+
# and the bytes will be gzipped.
47+
# ... also, we'll add .csv.gz on write.
48+
targ_wrap = Pipe(
49+
KeyCodecs.suffixed('.csv.gz'),
50+
ValueCodecs.csv() + ValueCodecs.str_to_bytes() + ValueCodecs.gzip()
51+
)
52+
53+
# Let's wrap our backends:
54+
src = src_wrap(src_backend)
55+
targ = targ_wrap(targ_backend)
56+
57+
# and copy src over to targ
58+
print(f"Before: {list(targ_backend)=}")
59+
targ.update(src)
60+
print(f"After: {list(targ_backend)=}")
61+
```
62+
63+
From the point of view of src and targ, you see the same thing.
64+
65+
```python
66+
assert list(src) == list(targ) == ['file_1', 'file_2']
67+
assert (
68+
src['file_1']
69+
== targ['file_1']
70+
== [['A', 'B', 'C'], ['one', 'two', 'three']]
71+
)
72+
```
73+
74+
But the backend of targ is different:
75+
76+
```python
77+
src_backend['file_1.pkl']
78+
# b'\x80\x04\x95\x19\x00\x00\x00\x00\x00\x00\x00]\x94(]\x94(K\x01K\x02K\x03e]\x94(K\x04K\x05K\x06ee.'
79+
targ_backend['file_1.csv.gz']
80+
# b'\x1f\x8b\x08\x00*YWe\x02\xff3\xd41\xd21\xe6\xe52\xd11\xd51\xe3\xe5\x02\x00)4\x83\x83\x0e\x00\x00\x00'
81+
```
82+
83+
Now that you've tested your setup with dictionaries, you're ready to move on to real,
84+
persisted storage. If you wanted to do this with local files, you'd:
85+
86+
```python
87+
from dol import Files
88+
src = Files('PATH_TO_LOCAL_SOURCE_FOLDER')
89+
targ = Files('PATH_TO_LOCAL_TARGET_FOLDER)
90+
```
91+
92+
But you could do this with AWS S3 using tools from
93+
[s3dol](https://github.com/i2mint/s3dol), or Azure using tools from
94+
[azuredol](https://github.com/i2mint/azuredol), or mongoDB with
95+
[mongodol](https://github.com/i2mint/mongodol),
96+
github with [hubcap](https://github.com/thorwhalen/hubcap), and so on...
97+
98+
All of these extensions provide adapters from various data sources/targets to the
99+
dict-like interface (called "Mapping" in python typing).
100+
What `dol` provides are base tools to make a path from these to the interface
101+
that makes sense for the domain, or business logic in front of you,
102+
so that you can purify your code from implementation details, and therefore be
103+
create more robust and flexible code as far as data operations are concerned.
104+
105+
106+
## Historical note
107+
18108
Note: This project started as [`py2store`](https://github.com/i2mint/py2store).
19109
`dol` is the core of py2store has now been factored out
20110
and many of the specialized data object layers moved to separate packages.

dol/tests/test_kv_codecs.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ def test_kvcodec_user_story_01():
4242
src_backend['file_1.pkl']
4343
# b'\x80\x04\x95\x19\x00\x00\x00\x00\x00\x00\x00]\x94(]\x94(K\x01K\x02K\x03e]\x94(K\x04K\x05K\x06ee.'
4444
targ_backend['file_1.csv.gz']
45-
# b'\x1f\x8b\x08\x00*YWe\x02\xff3\xd41\xd21\xe6\xe52\xd11\xd51\xe3\xe5\x02\x00)4\x83\x83\x0e\x00\x00\xtarg_backend['file_1.csv.gz']00'
45+
# b'\x1f\x8b\x08\x00*YWe\x02\xff3\xd41\xd21\xe6\xe52\xd11\xd51\xe3\xe5\x02\x00)4\x83\x83\x0e\x00\x00\x00'
4646

4747

4848
def _test_codec(codec, obj, encoded=None, decoded=None):

0 commit comments

Comments
 (0)