Support -Zno-link

[bjorn3]

Describe the problem you are trying to solve
Compilation pipelining can currently compile dependent crates as soon as the crate metadata is written in most cases. This improves cpu utilization for from-scratch builds. It is however not possible to apply pipelining on crates that need to be linked like executables and dylibs as all object files need to be available to link something.

Describe the solution you'd like
Using the -Zno-link option of rustc it is possible to separate the compilation and linking step. This makes it possible to pipeline the compilation and only wait for all crates to be fully compiled once we want to link the final executable. This option was introduced in rust-lang/rust#68487 and fixed in rust-lang/rust#80187 (now with a test to prevent regressions).

Notes
cc rust-lang/rust#64191

To implement this rustc will first need to be invoked with -Zno-link as argument. This will produce all object files and a .rlink file. Once all dependencies are compiled you need to invoke rustc a second time with -Zlink-only as argument and the .rlink file instead of the source file.

[weihanglo]

I am interested in this issue and plan to implement it. Just some questions need to make sure I didn't get it wrong.

Below is my understanding of a possible flow. @bjorn3, could you help me check its correctness?

1. The first rustc invocation is for all dependencies with -Zno-link.
2. And the next one is also for all dependencies with -Zlink-only.
3. Then compile and link the Target (crate root).
4. If there is any dependency attempting to produce a staticlib/dylib/proc-macro which requires upstream objects, the compilation pipeline must invoke rustc with -Zlink-only for all its upstream dependencies before proceeding.

Edit: Wrong. Was too young at that time though. See #9019 (comment).

@alexcrichton, since you are the person who implemented pipeline compilation and opened rust-lang/rust#64191, it would be very helpful if you can provide some advice and tips for prototyping. Thanks 😄

[bjorn3]

The first step is to compile all crates, with the crates that need to be linked getting an extra -Zno-link argument. Non-linked crates must not get -Zno-link as argument. This step can be fully pipelined, so once the metadata file of a dependency is written, all dependent crates can immediately be compiled without having to wait on codegen. This pipelining is unchanged from master except for also pipelining crates that need to be linked. The second step is after all crates are done compiling completely, only running rustc on the crates that need to be linked with -Zlink-only and with the .rmeta file from -Zno-link instead of the source file. If I am right, all crate types except for rlib need to be linked.

[alexcrichton]

This is likely to be a somewhat nontrivial change in Cargo. My best guess for the way to implement this would be to introduce a Unit for -Zno-link and a second unit for -Zlink-only for compilations that today are a single Unit which invoke the linker (e.g. everything other than an rlib basically). Cargo's unit of execution of a process is a Unit, and today there's a Unit-per-rustc-invocation basically. A Unit also tracks things like execution of a build script.

Pipelining with rlib dependencies is different since we don't invoke rustc twice. Instead we invoke rustc once and only once we've received the message that the rmeta is available do we start executing other units. There's special-casing that handles this (see JobQueue::enqueue and Artifact::Metadata), but if we're executing rustc twice then that probably doesn't need to happen.

I suspect functions like requires_upstream_objects and only_requires_rmeta may not need to be refactored, but you'll want to eye them carefully in case new support here impacts them.

[weihanglo]

Take another look of ii a few years after, I now have some doubts upon the benefit to compilation pipeline. For the real world crates, most of the dependencies are rlibs, not any other linkable artifacts. The most common linkable artifacts are bins from build scripts, which can't really be pipelined because their dependents still need to wait for build script executions. Given this, the benefit of splitting compile/link phase is more about configurability and cachability than pipelining.

That said, for crates depending on dylibs it does help, but it is a bit rare.

[bjorn3]

rustc_driver would benefit significantly from this. It is currently split into rustc_driver and rustc_driver_impl to effectively emulate -Zno-link/-Zlink-only.

[weihanglo]

Other future possibilities along with this change:

• We could offload linker flags from RUSTFLAGS to something like RUSTLINKFLAGS, so that normal rlib caches have a higher chance of being hit. And that would help a lot in per-user cache
• We might provide a way to opt-out, so people don't need caches go ahead and build everything all at once.

[bjorn3]

We could offload linker flags from RUSTFLAGS to something like #4349, so that normal rlib caches have a higher chance of being hit. And that would help a lot in #5931

Not as is, the -Zlink-only args need to match the -Zno-link invocation. And if you want to rerun -Zlink-only, you have to rerun -Zno-link to as the former removes all temporary files that the latter produced.

[weihanglo]

Is it a limitation of the current implementation, or the future possibility just doesn't make sense?

[bjorn3]

I think we could add it in the future, but it would require explicitly specifying which cli arguments are "link" arguments that can be set for -Zlink-only as well as moving a bunch of computations from the -Zno-link invocation to the -Zlink-only invocation. It should be doable, but it is not trivial. That said, I do think it would be nice to have in the future.