Misc Ideas for Enhancements

[thorwhalen]

To collect miscellaneous ideas for enhancement, new features, interfaces, etc.

[thorwhalen]

RAM-sensitive ZipReader

I'd like a ZipAccess(typing.Mapping) class in python that takes a zip file path.
It's __iter__ should give me an iterable of the filenames the zip contains, the __getitem__ should give me the bytes for that particular file, given the key.
Important though: I don't want to hold all the zips bytes in memory. I want the methods of my class to load only what they need to, and only temporarily.

import zipfile
from typing import Mapping

class ZipAccess:
    def __init__(self, zip_file_path):
        self.zip_file_path = zip_file_path

    def __iter__(self):
        with zipfile.ZipFile(self.zip_file_path, 'r') as zip_archive:
            return iter(zip_archive.namelist())

    def __getitem__(self, key):
        with zipfile.ZipFile(self.zip_file_path, 'r') as zip_archive:
            if key not in zip_archive.namelist():
                raise KeyError(f"'{key}' not found in the zip file.")
            with zip_archive.open(key) as file:
                return file.read()

# Example usage:
if __name__ == "__main__":
    zip_path = "example.zip"
    zip_access = ZipAccess(zip_path)

    # Iterating over filenames in the zip
    for filename in zip_access:
        print(filename)

    # Accessing the bytes of a specific file in the zip
    file_contents = zip_access['example.txt']
    print(file_contents.decode('utf-8'))  # Assuming it's a text file

[thorwhalen]

Clean up explicit tools

We have a module for "explicit" stores: dol/explicit.py.
By "explicit", I mean when the collection of keys (or values) are explicitly given.
It has always been a bit all over the place, though the theme that comes up a lot.

Note that a lot of the same functionalities can be taken care of, more or less, via wrap_kvs, so we may want to use these.

I'd propose to have a mini project where we analyze what's there, document and cover it with test it using AI, then try to replace with wrap_kvs-based code. Also, it would be nice to have some tutorial on this theme so that people (and AI) can better understand how to use it.

Things to check out

In the following discussion comment, I'm using wrap_kvs for some "explicit" mechanisms: Data acquisition stores

In config2py.base, there's a FuncBasedGettableContainer that wraps a Callable[[KT], VT] function so it has a (partial)
Mapping[KT, TT] interface

[thorwhalen]

Forwarding methods to extend mappings?

Given the "special methods" (__iter__, __len__, ...) get special treatment, which stops me from being able to simply change their definition "dynamically", which would make wrapping mappings a whole lot less verbose, I often thought that using non-special methods as the base, and simply forward the special methods to these might be a better idea.
I'm not yet sure if it's a good idea or not, but here are a few notes on that.

from collections.abc import Mapping

class MappingHooks(Mapping):
    def __iter__(self):
        return self.__iter()
    def __len__(self):
        return self.__len()
    def __contains__(self, key):
        return self.__contains(key)
    def __getitem__(self, key):
        return self.__getitem(key)
    
class ExampleMapping(MappingHooks, Mapping):
    """Just forwards all method calls to the wrapped dict"""
    def __init__(self, d: dict):
        self.d = d
    def __iter(self):
        return iter(self.d)
    def __len(self):
        return len(self.d)
    def __contains(self, key):
        return key in self.d
    def __getitem(self, key):
        return self.d[key]
    

m = ExampleMapping({'a': 1, 'b': 2})

assert list(m) == ['a', 'b']
assert len(m) == 2
assert 'a' in m
assert 'b' in m
assert 'c' not in m
assert m['a'] == 1
assert m['b'] == 2

Notes:

• Prefixing the names of the base methods with double-underscore seems to make it automatically treated as a special method. So for example, if you replace _iter with __iter (as I did origially) here, you'll get AttributeError: 'ExampleMapping' object has no attribute '_MappingHooks__iter'
• class ExampleMapping(Mapping, MappingHooks) leads to TypeError: Cannot create a consistent method resolution order (MRO) for bases Mapping, MappingHooks.
• Just writing __iter__ = _iter, __len__ = _len in a base class doesn't work since __iter__ is then bound to the particular _iter method (changing the _iter in a subclass doesn't have the effect expected.