Skip to content

[red-knot] Allow classes with __call__ method to be a subtype of Callable #17005

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 11 commits into from
Closed

[red-knot] Allow classes with __call__ method to be a subtype of Callable #17005

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
34f6824
Allow classes with __call__ method to be a subtype of Callable
MatthewMckee4 Mar 26, 2025
a343bb8
Update types.rs
MatthewMckee4 Mar 28, 2025
f2c748b
Update signatures.rs
MatthewMckee4 Mar 28, 2025
2309715
Fix some issues from review
MatthewMckee4 Mar 28, 2025
dbc5867
Merge branch 'class-with-call-subtype-callable' of https://github.com…
MatthewMckee4 Mar 28, 2025
dc2a5b0
Merge branch 'main' into class-with-call-subtype-callable
MatthewMckee4 Mar 28, 2025
ba1e58f
Fix branch error
MatthewMckee4 Mar 28, 2025
eed5e1a
Update crates/red_knot_python_semantic/resources/mdtest/type_properti…
MatthewMckee4 Mar 28, 2025
fa18160
Update crates/red_knot_python_semantic/src/types.rs
MatthewMckee4 Mar 28, 2025
8cd517c
Fix test file
MatthewMckee4 Mar 28, 2025
04cb547
Remove package-lock
MatthewMckee4 Mar 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions crates/red_knot_python_semantic/resources/mdtest/type_properties/is_subtype_of.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1053,5 +1053,22 @@ static_assert(is_subtype_of(TypeOf[C.foo], object))
static_assert(not is_subtype_of(object, TypeOf[C.foo]))
```

### Classes with __call__

#### Normal instance method

```py
from typing import Callable
from knot_extensions import TypeOf, is_subtype_of, static_assert, is_assignable_to

class A:
def __call__(self, a: int) -> int:
return a

static_assert(is_subtype_of(A, Callable[[int], int]))
static_assert(not is_subtype_of(A, Callable[[], int]))
static_assert(not is_subtype_of(Callable[[int], int], A))
```

[special case for float and complex]: https://typing.readthedocs.io/en/latest/spec/special-types.html#special-cases-for-float-and-complex
[typing documentation]: https://typing.readthedocs.io/en/latest/spec/concepts.html#subtype-supertype-and-type-equivalence
31 changes: 31 additions & 0 deletions crates/red_knot_python_semantic/src/types.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -710,6 +710,28 @@ impl<'db> Type<'db> {
self_subclass_ty.is_subtype_of(db, target_subclass_ty)
}

// An instance type with `__call__` method can be a subtype of Callable
(Type::Instance(InstanceType { class }), Type::Callable(_)) => {
let call_symbol = class.class_member(db, "__call__").symbol;
if call_symbol.is_unbound() {
false
} else {
match call_symbol {
Symbol::Type(Type::FunctionLiteral(call_function), _) => {
let callable_type =
call_function.into_callable_type_without_self_parameter(db);
match callable_type {
Type::Callable(CallableType::General(_)) => {
callable_type.is_subtype_of(db, target)
}
_ => false,
}
}
_ => false,
}
}
}

// `Literal[str]` is a subtype of `type` because the `str` class object is an instance of its metaclass `type`.
// `Literal[abc.ABC]` is a subtype of `abc.ABCMeta` because the `abc.ABC` class object
// is an instance of its metaclass `abc.ABCMeta`.
Expand Down Expand Up @@ -4135,6 +4157,15 @@ impl<'db> FunctionType<'db> {
)))
}

pub(crate) fn into_callable_type_without_self_parameter(self, db: &'db dyn Db) -> Type<'db> {
let signature = self.signature(db);
let new_parameters = signature.parameters().clone().without_self_parameter();
Type::Callable(CallableType::General(GeneralCallableType::new(
db,
Signature::new(new_parameters, signature.return_ty),
)))
}

/// Typed externally-visible signature for this function.
///
/// This is the signature as seen by external callers, possibly modified by decorators and/or
Expand Down
13 changes: 13 additions & 0 deletions crates/red_knot_python_semantic/src/types/signatures.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -493,6 +493,19 @@ impl<'db> Parameters<'db> {
.enumerate()
.rfind(|(_, parameter)| parameter.is_keyword_variadic())
}

/// Return a new Parameters type with the first parameter.
pub(crate) fn without_self_parameter(self) -> Self {
if let Some(first_param) = self.get(0) {
return if first_param.name().map(ruff_python_ast::name::Name::as_str) == Some("self") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not check the name of the parameter, it doesn't matter.

I think this method should return the removed parameter, because in the caller we should check if its annotated type (if any) is assignable from the instance type. That does matter. (We should add a test for the case where it isn't, e.g.

class C:
    def __call__(self: int) -> int:
        return 1

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How should that example be handled? It seems like this should be rejected before getting to this stage but i could be wrong.

pylance says "Type of parameter "self" must be a supertype of its class "A""
I guess since type of self is not yet dealt with, we cant do this.

I may need some more guidance on this anyway, what do you think of https://github.com/python/mypy/blob/b1be379f88e1e3735e04931f8253bbde4b387602/mypy/typeops.py#L366 and implementing a bind method (instead of into_callable_type_without_self_parameter which i think currently is not a very good function) to FunctionType that can take in the Class instance and return Self or Type

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call, thanks for pushing on whether this should be done in a better way. After looking at it, I think it should be done differently. We shouldn't need this function. Instead of using class_member to look up __call__, we should just use .member() on the instance type itself, and then call .signature() on the resulting type. In the typical case the .member() call will give us back a Type::Callable(BoundMethod(_)), and Type::signature already does the right thing for a bound method, including issuing a diagnostic in the case where self has a wrong type annotation.

I do think we'll probably want to also issue a diagnostic at the definition site of a method with a bad self annotation, but that's really separate and should be handled separately. Notice that pyright issues diagnostics in both places (both the definition site of __call__, and the site where it is "used"). We are just worrying about the latter in this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

(Type::Instance(_), Type::Callable(_)) => {
                let call_symbol = self.member(db, "__call__").symbol;
                if call_symbol.is_unbound() {
                    false
                } else {
                    match call_symbol {
                        Symbol::Type(
                            Type::Callable(CallableType::BoundMethod(call_function)),
                            _,
                        ) => {
                            let callable_type = call_function.function(db).into_callable_type(db);
                            match callable_type {
                                Type::Callable(CallableType::General(_)) => {
                                    callable_type.is_subtype_of(db, target)
                                }
                                _ => false,
                            }
                        }
                        _ => false,
                    }
                }
            }

into_callable_type here still give a GeneralCallableType with the self parameter, it seems like maybe BoundMethodType should handle removing self from the parameters of its function?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No we should not be matching on the specific type of the symbol result like that and pulling out internal details of the type. Instead we should just call .signature() on the type (no matter what it is) and construct a general callable type from that signature, and check if that's a subtype of target. Type::signature already takes care of correctly handling the self parameter. The resulting code should be quite a bit simpler and more general than what you pasted above. (And as a bonus it will handle all kinds of edge cases better where __call__ is something other than a normal method, too).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(We also don't need the separate if test for is_unbound(), that can just fall into the default case of the match, since it's not Symbol::Type.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized this should go one step further: Type::signatures already has handling for looking up __call__ on a Type::Instance, so we don't even have to do that here, we can just call .signatures(db) directly on the Instance type.

There is some complexity about the fact that GeneralCallableType only holds a single Signature, but if you get a more complex Signatures back from the instance type, you can just punt on that with a TODO for now.

Copy link
Contributor Author

@MatthewMckee4 MatthewMckee4 Mar 28, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

            // An instance type with `__call__` method can be a subtype of Callable
            (Type::Instance(_), Type::Callable(_)) => {
                let signatures = self.signatures(db);
                let mut iter = signatures.iter();
                match iter.next() {
                    Some(signature) => {
                        let mut overloads = signature.iter();
                        match overloads.next() {
                            Some(overload) => {
                                let callable_type = Type::Callable(CallableType::General(
                                    GeneralCallableType::new(db, overload),
                                ));
                                callable_type.is_subtype_of(db, target)
                            }
                            None => false,
                        }
                    }
                    None => false,
                }
            }

I may be on the wrong track here, but callable_type here still has a self parameter

Copy link
Contributor

@carljm carljm Mar 28, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, you're on the right track. It looks like the issue here is that the way we handle "binding self" for a bound method is not by removing the parameter, but by adding a bound_type to the CallableSignature, and then when we call it we know to bind that bound_type to the first parameter, and bind the provided arguments to the rest of the parameters. I think this is the right approach, as it lets us check type assignability for self.

The problem you're hitting here is that we set bound_type on the CallableSignature (which represents potentially a set of overloads), but GeneralCallableType only wraps a single Signature, which doesn't carry that bound_type.

I think really we need to resolve this issue in the context of broader support for overloads; we need to add support for callable subtyping with overloads, and with bound_type, and we don't have that yet. Probably ultimately GeneralCallableType should wrap a CallableSignature, not just a single Signature. So I guess proper resolution of this issue is a bigger task than we'd realized. It may be best to just hold off on it for now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay sounds good. Thank you for the help anyway, i really appreciate it.

let new_parameters = self.as_slice()[1..].to_vec();
Parameters::new(new_parameters)
} else {
self
};
}
self
}
}

impl<'db, 'a> IntoIterator for &'a Parameters<'db> {
Expand Down
Loading