Skip to content

Conversation

xclaesse
Copy link
Member

If the Cargo subproject is part of the main repo we had to write a dummy
wrap file to specify the method:

[wrap-file]
method=cargo

We can now instead directly call subproject('rust', method: 'cargo')
with no wrap file. This also allows the method to be a user
option: method: get_option('cargo') ? 'cargo' : 'meson'.

It was wasteful, but also failed when the .wrap file miss that
dependency from [provide] section and no Cargo.lock file provide it
either.
@bonzini
Copy link
Collaborator

bonzini commented Oct 19, 2025

First patch looks good, second should wait for #12057 and a review from other committers.

If the Cargo subproject is part of the main repo we had to write a dummy
wrap file to specify the method:
```
[wrap-file]
method=cargo
```

We can now instead directly call `subproject('rust', method: 'cargo')`
with no wrap file. This also allows the method to be a user
option: `method: get_option('cargo') ? 'cargo' : 'meson'`.
@dcbaker
Copy link
Member

dcbaker commented Oct 20, 2025

I merged the linked PR.

I have another use case for this that came up today, actually. I have a project that has vendored copies of dependencies patched with CMake build support, but I want to use the versions from the wrapdb for the meson support. The result is that Meson errors out with "two subprojects provide the same dependency". Even if this doesn't get us all the way there, it should get us most of the way there.

@eli-schwartz
Copy link
Member

I have another use case for this that came up today, actually. I have a project that has vendored copies of dependencies patched with CMake build support, but I want to use the versions from the wrapdb for the meson support. The result is that Meson errors out with "two subprojects provide the same dependency". Even if this doesn't get us all the way there, it should get us most of the way there.

Shouldn't we be handling this on a much more fundamental(lying reliable) level by preferring the top level project? And I believe @jpakkane also said this shouldn't be happening anyway because subproject wrap promotion should be an explicit choice by the top level project. :) If we are hitting these sorts of conflicts in the wild then that makes a compelling argument that... @jpakkane is right and we should never have implicitly started recursively loading anything?

...

More generally I can understand the desire to avoid adding a wrap file but I don't think it's a very strong reason, and I'm nervous about weakening the existing dependency kwarg.

@dcbaker
Copy link
Member

dcbaker commented Oct 20, 2025

The problem I have is that I have a layout like this:

subprojects/
   snappy/<patched version with cmake support>
   google-snappy.wrap

Meson tries to load both of these, and then says "You have two versions of snappy! I quit!"

But I can't delete the snappy/ version becuase the CMake build needs it, but I also would rather use a version with native meson build definitions than rely on the CMake dependency fallback.

@xclaesse
Copy link
Member Author

@dcbaker your plan is to allow multiple subproject with the same name, but then select the one that supports the method you want? That seems a bit sketchy to me 😕

We are getting off topic, but I've always wanted to add some config file in e.g. subprojects/wraps.ini where we could have an exclude list to ignore some local directories.

@xclaesse
Copy link
Member Author

Shouldn't we be handling this on a much more fundamental(lying reliable) level by preferring the top level project?

We do that already of course. The conflict happens only if the same project has 2 wraps that provide the same dep.

@eli-schwartz
Copy link
Member

Okay reading the diff now, I see it is a new kwarg that is different between subproject() and dependency(), which I suppose is what I would recommend if I wanted this functionality in some form. I still don't really like this change.

The proposed reasoning for wanting this is really cargo specific, imo. And there are good reason for why it's cargo specific -- I'll elaborate on those in a minute.

The other argument in favor basically says that this doesn't actually fix anything whatsoever, but indirectly can be used as a kind of excuse to add an unrelated feature, that being use of the method as a backdoor to disambiguate multiple conflicting wrap definitions. We shouldn't have conflicts within a given level and we shouldn't allow subordinate projects to hijack a wrap that the main project, which is fundamentally in charge, decided on. And disambiguating via the method is a really crude way of doing so anyway, because it assumes (blindly) that you won't have the problem of both projects trying to provide a cmake wrap!

I have big issues with allowing CMake to be specified here at all. It is conceptually nonsense to specify CMake as the fallback method because CMake does not and cannot do override_dependency anyway. @xclaesse didn't we have this conversation when initially adding wrap method support, already? I view CMake here as effectively dead in the water: you need a wrap file.

Even if you didn't need a wrap file, such as for subproject('foo', fallback_method: 'cmake'), which allows you to access properties of the subproject rather than working via dependency overrides, this duplicates the exact functionality of the import('cmake') module. We specifically market this as the way to fiddle with cmake, the return value Holder has drastically different methods, and you need it in order to pass options which wrap files cannot do (as per the previous discussion about wrap methods).

It feels like we've sneakily moved the Overton window on what's considered a good way to interact with subprojects. That makes me wonder if I should have pushed back harder on objecting to adding wrap methods at all, if we will need to continually relitigate these sorts of decisions.

Anyway. Cargo wraps. I actually think this is fine, and doesn't need "fallback" methods, because it seems very analogous to how dependency('foo', method: 'dub') works. It looks up a dependency in a specific programming language and that programming language's ecosystem, there's no possibility of using a different dependency detector (unlike cmake wraps which usually can be looked up via CMake dependencies or pkg-config).

dependency('foo', method: 'cargo') would be something I expect to detect what we have up until now called dependency('foo-rs') and sidejacked into fallbacks. If we have this, we don't need name mangling, do we?

@xclaesse
Copy link
Member Author

I see it is a new kwarg that is different between subproject() and dependency(), which I suppose is what I would recommend if I wanted this functionality in some form.

That's a part I don't like, but unfortunately method is already taken for dependency(). Hijacking the existing kwarg is tempting and kind of make sense for cargo indeed. But I fear it creates even more confusion.

Even if you didn't need a wrap file, such as for subproject('foo', fallback_method: 'cmake'), which allows you to access properties of the subproject rather than working via dependency overrides, this duplicates the exact functionality of the import('cmake') module.

I've always been against cmake.subproject() because it bypass the whole dependency fallback mechanism, which makes the whole thing pretty useless. To be fair I've always been against the whole cmake subproject stuff, it should be part of external_project module. But that's another story.

Now you have a good point that cmake subproject does not seems to be using override_dependency(). Maybe it predates it? I don't see why it couldn't do it. Thanks, now I hate that code even more :D

@xclaesse
Copy link
Member Author

dependency('foo', method: 'cargo') would be something I expect to detect what we have up until now called dependency('foo-rs') and sidejacked into fallbacks. If we have this, we don't need name mangling, do we?

You still need to tell which API level you want, you can have both foo-1-rs and foo-2-rs. We could in theory drop the -rs suffix when method=cargo, not sure it's worth it.

@eli-schwartz
Copy link
Member

Even if you didn't need a wrap file, such as for subproject('foo', fallback_method: 'cmake'), which allows you to access properties of the subproject rather than working via dependency overrides, this duplicates the exact functionality of the import('cmake') module.

I've always been against cmake.subproject() because it bypass the whole dependency fallback mechanism, which makes the whole thing pretty useless. To be fair I've always been against the whole cmake subproject stuff, it should be part of external_project module. But that's another story.

We added wrap method=cmake support specifically because you pushed for it! :)

It is capable today of handling the straightforward cases. There's still cases that aren't straightforward and require the flexibility of the module.

And my personal belief is that I am diametrically opposed to permitting a CMakeSubprojectHolder to be returned by subproject(). It is a horrible UX. But dependency() exports a consistent API regardless of what kind of project it came from -- it is "just" a declare_dependency, even if it came from cmake.

Now you have a good point that cmake subproject does not seems to be using override_dependency(). Maybe it predates it? I don't see why it couldn't do it. Thanks, now I hate that code even more :D

Can I refresh your memory? :) #11730 (comment)

It doesn't predate it so much as CMake itself has oppositional defiance disorder with regard to design of build systems, and that specifically conflicts with the kind of guarantees we would like and which dependency overrides require. To wit: dependencies have different names compared to their target names, because dependencies are actually literally just prose documentation about which internal variable to use, the variables will NOT be named the same as either the project or the library or the pkg-config dependency name, and you are simply supposed to not use the other variables at all, even (or especially if) they are internal static libraries that conflict with the name of system dependencies.

@bonzini
Copy link
Collaborator

bonzini commented Oct 21, 2025

dependency('foo', method: 'cargo') would be something I expect to detect what we have up until now called dependency('foo-rs') and sidejacked into fallbacks.

The way I would like to spell it is

    rust = import('rust')
    cargo = rust.workspace()
    foo_rs = cargo.subproject('foo'[, '1']).dependency([rust_abi: 'c'|'rust'])

This means that you're forced to invoke rust.workspace() (and trigger global resolution) before using a Cargo subproject, which makes sense since the resolution depends on the Cargo.lock/Cargo.toml file.

In its current state #15069 doesn't include subproject but it would be easy to add it.

@dcbaker
Copy link
Member

dcbaker commented Oct 21, 2025

@dcbaker your plan is to allow multiple subproject with the same name, but then select the one that supports the method you want? That seems a bit sketchy to me 😕

No, I only want to support the native Meson one because the CMake one doesn't work well for Meson. But the project in question has a CMake build system that will not be removed, and I need to not destroy the CMake build, but make the Meson one work. So what I want is:

dependency('snappy', fallback_method : 'meson')

I want to never use the CMake one. More like dependency('LLVM', method : 'config-tool')

@eli-schwartz
Copy link
Member

eli-schwartz commented Oct 21, 2025

This explanation does not compute for me.

The problem I have is that I have a layout like this:

subprojects/
   snappy/<patched version with cmake support>
   google-snappy.wrap

Meson tries to load both of these, [..]

but you think the solution to meson seeing a directory named AAA that isn't intended to be parsed as an available wrap, and has a genuine BBB.wrap file referencing the directory name AAA but containing required information and overriding [provide] AAA = AAA_dep, is...

to invoke BBB as dependency('AAA', fallback_method: 'meson')?

Not only don't I understand why this is supposed to be a good idea, I also don't understand how this PR is even supposed to represent a step towards fixing the problem. I am now more convinced than I was before your explanation, that this isn't a good solution to the problem you believe you have.

I'd also quite like to know why nobody else has that problem (of having a directory named snappy/ instead of a directory named snappy-1.2.2/) but I'm happy to concede that loading wraps implicitly from their directory name when they are already referenced by a wrap file (???) is a bug regardless of the WrapDB definition for google-snappy.wrap

@dcbaker
Copy link
Member

dcbaker commented Oct 21, 2025

Not only don't I understand why this is supposed to be a good idea, I also don't understand how this PR is even supposed to represent a step towards fixing the problem. I am now more convinced than I was before your explanation, that this isn't a good solution to the problem you believe you have.

snappy/ is a subdirectory in the subdirectory for subprojects that is provided for the CMake build (called thirdparty, as CMake projects tend to do). It is a git submodule pointing to a version of snappy that has various patches applied to it, including force a static library.

Then I've added google-snappy.wrap, because the snappy in said subdirectory is not suitable for Meson. Meson sees this directory, it also sees the wrap, it decides that there are two providers for snappy, and errors out. I simultaneously do not want to use the snappy subdir (I want to honor default_library), and I cannot delete it because some users are unwilling or unable to use Meson, and I cannot rename it, because CMake users find that confusing. I have also been given a requirement that all third party code must live in this /thirdparty directory, so no subprojects/ directory that would separate these would be allowed.

So right now, there is no way to use the snappy wrap because a subdirectory called snappy exists, which cannot be removed, and which cannot be renamed, and which would be inferior to the warp for my purposes. I absolutely see a keyword that says "Only try a meson subproject and just ignore that cmake thing there" as absolutely solving my problem in the same way the method keyword that says "Hey, ignore cmake and only use the config-tool to find this dependency" solves the problem of "The CMake finder is inferior in some way to the config-tool"

@bonzini
Copy link
Collaborator

bonzini commented Oct 21, 2025

there is no way to use the snappy wrap because a subdirectory called snappy exists, which cannot be removed, and which cannot be renamed

That would be fixed by letting a wrap always win over a subdirectory.

@dcbaker
Copy link
Member

dcbaker commented Oct 21, 2025

Sure

@bonzini
Copy link
Collaborator

bonzini commented Oct 22, 2025

#15158 now is in a semi-mergeable state, and implements cargo.subproject('num-complex').dependency().

@eli-schwartz
Copy link
Member

Then I've added google-snappy.wrap, because the snappy in said subdirectory is not suitable for Meson. Meson sees this directory, it also sees the wrap, it decides that there are two providers for snappy, and errors out. I simultaneously do not want to use the snappy subdir (I want to honor default_library), and I cannot delete it because some users are unwilling or unable to use Meson, and I cannot rename it, because CMake users find that confusing. I have also been given a requirement that all third party code must live in this /thirdparty directory, so no subprojects/ directory that would separate these would be allowed.

So right now, there is no way to use the snappy wrap because a subdirectory called snappy exists, which cannot be removed, and which cannot be renamed, and which would be inferior to the warp for my purposes. I absolutely see a keyword that says "Only try a meson subproject and just ignore that cmake thing there" as absolutely solving my problem in the same way the method keyword that says "Hey, ignore cmake and only use the config-tool to find this dependency" solves the problem of "The CMake finder is inferior in some way to the config-tool"

I don't consider it an acceptable solution to this problem, to require magic prompts in meson.build that may mess up if first called in a subproject and divide configuration between the wrap file and meson.build

The core problem you have is NOT that you want to select which method to use to try to build the subproject. The core problem you have is that it is an error and a fallacy for meson to misdetect one of the two as being a distinct standalone component when it is actually an implementation detail of the other (?). That is very different from a system pkg-config file and a system foo-config.cmake that each exist independently and don't relate to each other at all, and may have unique pros and cons.

Hence @bonzini's suggestion was also my belief: we shouldn't be loading a directory as an implicit wrap, if an explicit *.wrap claims to own that directory. It's a "simple" bugfix, and doesn't require projects to modify their meson.build (or their wrap file) at all. That's good -- we can make something magically work automatically.

Adding new meson.build syntax instead, would be totally backwards. So if this was the only reason to add a new syntax, I would 100% object to the new syntax. And if there are other reasons to add a new syntax (and this PR was in fact about a totally different reason to add a new syntax) I want to exclusively discuss that other reason, rather than this reason that seems objectively bad.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants