Runtime Access to Generic Types on Class Method

[saulshanabrook]

I would like to be able to access the type arguments of a class from its class methods.

To make this concrete, here is a generic instance:

import typing
import typing_inspect

T = typing.TypeVar("T")

class Inst(typing.Generic[T]):
    @classmethod
    def hi(cls):
        return cls

I can get its generic args:

>>> typing_inspect.get_args(Inst[int])
(int,)

But, these seem to get erased inside the class methods:

>>> typing_inspect.get_args(Inst[int].hi())
()

[saulshanabrook]

So the issue is that descriptors bound on generic classes that are parametrized are called on the base class not parameterized class:

This is a reduced version of the above:

import typing
import typing_inspect

T = typing.TypeVar("T")

class MyDescriptor:
    def __get__(self, obj, objtype):
        return objtype


class MyClass(typing.Generic[T]):
    x = MyDescriptor()

c = MyClass[int] 

print(c)
# __main__.MyClass[int]
print(c.x)
# __main__.MyClass

I guess the issue here is maybe that x is not in cs dict:

>>> c.__dict__
{'_inst': True,
 '_special': False,
 '_name': None,
 '__origin__': __main__.MyClass,
 '__args__': (int,),
 '__parameters__': (),
 '__slots__': None,
 '__module__': '__main__'}

So it will look it up on it's base, which is MyClass instead of MyClass[int].

[ilevkivskyi]

As I mentioned off-line this is not something easy to fix. We need some time to figure out possible
(realistic) options.

[saulshanabrook]

As a workaround, I was able to monkey patch the __getattr__ on the generic alias so that it checks if the user is trying to access a descriptor and if so, use itself as the class instead of the origin class:

import typing


def generic_getattr(self, attr):
    """
    Allows classmethods to get generic types
    by checking if we are getting a descriptor type
    and if we are, we pass in the generic type as the class
    instead of the origin type.

    Modified from
    https://github.com/python/cpython/blob/aa73841a8fdded4a462d045d1eb03899cbeecd65/Lib/typing.py#L694-L699
    """

    if "__origin__" in self.__dict__ and not typing._is_dunder(attr):  # type: ignore
        # If the attribute is a descriptor, pass in the generic class
        property = self.__origin__.__getattribute__(self.__origin__, attr)
        if hasattr(property, "__get__"):
            return property.__get__(None, self)
        # Otherwise, just resolve it normally
        return getattr(self.__origin__, attr)
    raise AttributeError(attr)

typing._GenericAlias.__getattr__ = generic_getattr  # type: ignore

With this change, both of the examples above work as expected.

[JukkaL]

In my opinion this is not something we should support even if it was technically possible. Type annotations are primarily for static checking, not for runtime purposes. Any runtime uses of types beyond very simple ones will likely hit all sorts of limitations soon enough.

[saulshanabrook]

Type annotations are primarily for static checking, not for runtime purposes.

Where is the right venue to have a conversation about supporting different runtime purposes for type hints? Would drafting a PEP to articulate a possible runtime API for using typing annotations be helpful? Or should I start a discussion on python-ideas or on discourse?

For my use case, in metadsl, I need to be able to compute the return type of a function given some arguments. I am able to do this currently using the existing runtime hooks, but it would be better if I knew I was building on solid APIs for this. I would be happy to articulate as well why analyzing the type hints led to a simpler UX in this library, I chatted with @msullivan a bit at PyCon about my use case and its relationship to mypyc (mine is at runtime to generic backends, where as mypyc is AOT to C).

[JukkaL]

@saulshanabrook The typing-sig@ mailing list a focused on discussing improvements to Python static typing. However, I suspect that most of the existing subscribers are primarily interested in static approaches. Note that mypyc also arguably mostly uses types statically (that is, during compilation). At runtime the types are erased to a quite simple form, much simpler than full PEP 484 types.

As with many other ideas, if you can demonstrate that your approach is popular among Python developers, it will be an easier sell. There are a lot of possible improvements to Python static typing and I believe that it's much easier get ideas accepted when the practical benefit relative to the complexity of the change is clear and easy to justify.

[gvanrossum]

I think the starting point should be a discussion on python-dev.

On Mon, May 20, 2019 at 07:48 Saul Shanabrook ***@***.***> wrote: Type annotations are primarily for static checking, not for runtime purposes. Where is the right venue to have a conversation about supporting different runtime purposes for type hints? Would drafting a PEP to articulate a possible runtime API for using typing annotations be helpful? Or should I start a discussion on python-ideas or on discourse? For my use case, in metadsl <https://github.com/Quansight-Labs/metadsl>, I need to be able to compute the return type of a function given some arguments. I am able to do this currently using the existing runtime hooks, but it would be better if I knew I was building on solid APIs for this. I would be happy to articulate as well why analyzing the type hints led to a simpler UX in this library, I chatted with @msullivan <https://github.com/msullivan> a bit at PyCon about my use case and its relationship to mypyc (mine is at runtime to generic backends, where as mypyc is AOT to C). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#629?email_source=notifications&email_token=AAWCWMXKSKC6X42ESIGXMFDPWK2ZNA5CNFSM4HKZC4EKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVZCEWY#issuecomment-494019163>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAWCWMTXQJU7OBG4DYEIMLTPWK2ZNANCNFSM4HKZC4EA> .

-- --Guido (mobile)

[ilevkivskyi]

@saulshanabrook One of the problems that I see (apart from maintenance burned) is that every way of supporting this I can imagine causes some performance penalty for people who will not use this (and this module is used by a lot of people). Have you tried to perform any benchmarks for various operations with the patched __getattr__ that you propose?

[ilevkivskyi]

#616 is another example where this may be useful.

[dudil]

In my opinion this is not something we should support even if it was technically possible. Type annotations are primarily for static checking, not for runtime purposes. Any runtime uses of types beyond very simple ones will likely hit all sorts of limitations soon enough.

Hi @JukkaL - just a comment as a user. Typing is very useful to anyone who is creating extensive projects in Python. It allows better readability of abstract class and code reuse (when used right).
I guess these are among the reasons why they were added in the first place.
[Sorry for not going to any debate if python is the platform of choice for extensive projects - don't think this is relevant]
Now - in the case of generic, the main issue is that it does not support any public execution time interface for getting its instance type or inheritance while the same feature is supported for regular objects. (via issubclass and isinstance). This put typing class in a very much inferior place vs. regular objects.
Yes - I'm aware I could user origin and args but they are very much limited and not common for use.
Anyway - just a user comment here for your thoughts.

[Artemis21]

Type annotations are primarily for static checking, not for runtime purposes.

Many popular libraries use type annotations for things like parameter conversion - for example, Discord.py, FastAPI or Pydantic. I think this is a great way of reducing repeated code for such things.

[gpshead]

The typing-sig@ mailing list a focused on discussing improvements to Python static typing.

Recommendation for all: Join the list and make this not be true.

That statement is merely one of who had been active on the list because we needed to have multiple static checkers coordinate somewhere. Given recent discussions, everyone using Python type annotations should use it as a common place to centralize on needs.

[rickeylev]

I ran into this too while trying to port some code from 3.6 to 3.7. As an alternative to monkey patching GenericAlias (affects everything, which has its own problems), I put together some code so that a class can more localize the effect. This isn't well tested, but seems to more-or-less work.

import typing

class Proxy:
  def __init__(self, generic):
    object.__setattr__(self, '_generic', generic)

  def __getattr__(self, name):
    if typing._is_dunder(name):
      return getattr(self._generic, name)
    origin = self._generic.__origin__
    obj = getattr(origin, name)
    if inspect.ismethod(obj) and isinstance(obj.__self__, type):
      return lambda *a, **kw: obj.__func__(self, *a, *kw)
    else:
      return obj

  def __setattr__(self, name, value):
    return setattr(self._generic, name, value)

  def __call__(self, *args, **kwargs):
    return self._generic.__call__(*args, **kwargs)

  def __repr__(self):
    return f'<{self.__class__.__name__} of {self._generic!r}>'

class RuntimeGeneric:
  def __class_getitem__(cls, key):
    generic = super().__class_getitem__(key)
    if getattr(generic, '__origin__', None):
      return Proxy(generic)
    else:
      return generic

from typing import Generic, TypeVar
T = TypeVar('T')
class Usage(RuntimeGeneric, Generic[T]):
  @classmethod
  def foo(cls):
    print(cls.__args__, cls.__origin__)

This is still, fundamentally, hacky. It's a bit weird that cls isn't actually the runtime type, but a _GenericAlias-proxy-like-thing that has subtle differences (e.g. isinstance/issubclass behavior). It'd be cleaner if a class had a way to opt-in into preserving the generics information about itself in classmethods so that runtime introspection was more obvious/intuitive.

[pbarker]

Any update here? I'm also finding this behavior challenging and unexpected.

It'd be cleaner if a class had a way to opt-in into preserving the generics information about itself in classmethods so that runtime introspection was more obvious/intuitive.

100% agree

Are there threads folks have started on the mailing lists that we could link here?

[zwergziege]

There was an initial post at typing-sig, it didn't attract much attention. It can be found here: https://mail.python.org/archives/list/[email protected]/thread/T7VEN5HYHIT5ABNJHYOW434JHELTTKT3

[kurtbrose]

Reading this discussion with great interest. It's very nice and intuitive to handle structured data entering and exiting the system with dataclasses. For example, ORMs, network servers and clients, loading / saving to files.

I've been frustrated a few times trying to make this pattern work:

class Resource(Generic[DataType]):
   @classmethod
   def fetch(cls) -> list[DataType]:
       raw = http_get()
       data_cls = get_args(cls)[0]
       return [data_cls(**item) for item in raw["response"]]

The parameter ends up needing to be passed twice: Resource[ResponseData](ResponseData). (This is a pattern I've also encountered in Java as a work-around for generic type erasure).

This seems like a natural extension of TypeVar -- it's easy to write

def fetch(data_type: type[DataType]) -> DataType:
    ...

It's unexpected to hit a wall when you want to bundle a group of these functions together into a class.