type_caster_generic: add cast_sources abstraction

[oremanj]

Description

This is a pure refactor (no observable behavior changes) that centralizes the logic for choosing the C++ source object and type to use for a to-Python conversion. It replaces the old src_and_type() function (also written by yours truly, some eight years ago) that had some cumbersome interactions with smart_holder. It anticipates changes that will be needed in #5800 to provide more information to type_caster_generic::cast() for it to use when casting to a foreign type; such information will be carried in the cast_sources object. With cast_sources, the downcasting and type lookup behavior can be separated from the call to cast(), so it becomes possible to insert custom logic before or after the actual cast, which is especially useful when casting to foreign via a holder.

[rwgk]

The refactor is awesome!

My comments are purely about naming, please feel free to disregard, but I believe giving a few minutes of thought to intuitive (less generic) naming could help future maintainers and potential new contributors a lot to navigate the pybind11 code more easily.

[rwgk]

Awesome, thanks!

[PhilipDeegan]

It seems since the addition of this I am no longer able to compile.

I get a

pybind11/include/pybind11/detail/type_caster_base.h:1637:20: error: call of overloaded ‘cast(PHARE::pydata::PatchData<std::vector<double>, 1>*, pybind11::return_value_policy, pybind11::handle&)’ is ambiguous
 1637 |         return cast(std::addressof(src), return_value_policy::move, parent);

It's entirely possible we're using pybind wrong of course, but it was compiling

from a cursory glance, it seems I cannot have a something like this

template<typename T>
struct A{};

template<typename T, typename Y>
struct B{

     auto getA(){
         return A<T>{};
     }
}

class declaration

    py::class_<A<double>, py::smart_holder>(m, "ad");

    py::class_<B<double, double>, py::smart_holder>(m, "bdd")
         .def("getA", &B<double, double>::getA);
    py::class_<B<double, float>, py::smart_holder>(m, "bdf")
         .def("getA", &B<double, float>::getA);

[oremanj]

@PhilipDeegan I am unable to reproduce this compiler error using your example code. I somewhat doubt that you actually tested using your example code, because it doesn't compile (it's missing a semicolon after the definition of B). But if I fix that, it seems to compile just fine.

Please provide the entire compiler error. There should be quite a bit more after the two lines you pasted. There is probably some relevant content before those lines as well.

What compiler version and operating system are you running on?

Do you have any local customizations to pybind11 in your project?

It would be very helpful if you can provide a full example (not a sketch) that you have confirmed reproduces the problem.

[PhilipDeegan]

@oremanj thanks for the reply our project is somewhat complex so I was trying to minimize the overhead

What compiler version and operating system are you running on?

several, locally I have gcc 14 on debian 13 and our CI is clang 19 on Fedora 41, and both seem to show this

full error log from our CI is here

the stack in our code is from here

the function call specifically is this

and the return type is a vector of this

Do you have any local customizations to pybind11 in your project?

not sure what this means

to add, I tried switching from py::class_<T, std::shared_ptr<T>> to py::class_<T, py::smart_holder> and it seems to give a different ambiguous call error

[oremanj]

full error log from our CI is here

This error appears to be from a different compiler than the error you pasted two lines of. Could you provide the rest of the error that you pasted two lines of? It seems to have more detail.

Is there any way I could build your project locally? I see a github link so it's presumably open-source.

Is the failing CI a Github Actions job that you could link to?

not sure what this means

Are you using pybind11 completely unmodified as it exists in this repository, or do you have your own copy of it with some things changed?

to add, I tried switching from py::class_<T, std::shared_ptr> to py::class_<T, py::smart_holder> and it seems to give a different ambiguous call error

That would surprise me, but this whole thing surprises me. Can you provide that error as well?

[oremanj]

I think I see the problem. You've created a class that is implicitly convertible from anything at all, including a pointer to itself. This PR added an implicit conversion in the call to cast(PatchData*). The conversions to PatchData& and to cast_sources& are equally good.

    template<typename... Args>
    PatchData(Args&&... args)
        : data{std::forward<Args...>(args...)}
    {
    }

From a C++ style perspective, this constructor should either be explicit or should be constrained to only accept Args that are suitable for constructing the Data.

This is an easy fix in pybind also though. I'll put up a PR shortly.

[PhilipDeegan]

oh maybe, you're thinking this?

requires std::is_constructible_v<Data, Args&&...> ?

[oremanj]

Yes.

[PhilipDeegan]

I confirm it works, cheers!

[oremanj]

Posted #5873 which should fix this on the pybind side.