Nightly builds of our packages depend on nightlies of their dependencies, both within and across repositories. For example, cuml 26.08.00 alphas depend on both libcuml 26.08.00 alphas and cuvs/raft/rmm nightlies. In the past, we have encountered numerous issues where the lack of a tighter pinning would result in installing incompatible builds into test environments and resulting in failures that would be opaque to debug. Since usage of APIs across libraries is more limited, and especially so at the C++ layer, the biggest problems were typically from installing incompatible versions of the Python and C++ libraries from the same repository, e.g. cudf and libcudf mismatches.
To ameliorate this situation, we introduced a new versioning strategy where every package build would be assigned a monotonically increasing alpha version and that same alpha version would be assigned consistently and pinned between different components, so cudf 26.08.00a100 would depend exactly on libcudf 26.08.00a100. That fixed many of our issues.
Now, we are interested in finding ways to reduce CI times, and one of the options is pruning build jobs as well as test jobs based on what files are changed in the PR. For example, a PR that only modifies the pure Python components of a package should not require rebuilds of either C++ or Cython components. The main blocker to this right now is that the versioning scheme developed above prohibits a mismatched alpha version.
We should be able to fix this by instead building against the most recent version of the dependencies that has a build already finished and uploaded to our nightly channels/indexes. For PRs where no C++ code was changed, the C++ build jobs could be skipped, and the Python jobs could search for the most recent completed C++ build, get the alpha version, and inject that as the build dependency. Then there would still be a tight pinning, avoiding the problems discussed above, but the pinning could be to an older alpha version that is API and ABI compatible, reducing the amount of build duplication needed.
This approach does introduce a few new failure modes that we must consider:
- If a PR is behind main, it could be missing updates to the Cython/Python code that are necessary for compatibility with the underlying C++ library. Developers would have to recognize this mode of failure and merge the latest main to overcome it. Given that we already have cases where such a merge is required (e.g. if another PR is made to unblock CI by skipping or fixing some tests), I think this is not a huge issue.
- When a PR making a C++ API-breaking change merges, no other PRs will be compatible with that until the corresponding build job completes. That will introduce an additional delay. I think that is probably acceptable since it drops the E2E time, but we may have to experiment with this approach for a while to know for sure.
- If two PRs making breaking changes occur simultaneously, we may need to manually trigger a build job to produce a suitable dependency package. I could see this getting a bit tedious. Again, we may have to try and see if the payoff in reduced per-PR CI time is worthwhile.
Nightly builds of our packages depend on nightlies of their dependencies, both within and across repositories. For example, cuml 26.08.00 alphas depend on both libcuml 26.08.00 alphas and cuvs/raft/rmm nightlies. In the past, we have encountered numerous issues where the lack of a tighter pinning would result in installing incompatible builds into test environments and resulting in failures that would be opaque to debug. Since usage of APIs across libraries is more limited, and especially so at the C++ layer, the biggest problems were typically from installing incompatible versions of the Python and C++ libraries from the same repository, e.g. cudf and libcudf mismatches.
To ameliorate this situation, we introduced a new versioning strategy where every package build would be assigned a monotonically increasing alpha version and that same alpha version would be assigned consistently and pinned between different components, so cudf 26.08.00a100 would depend exactly on libcudf 26.08.00a100. That fixed many of our issues.
Now, we are interested in finding ways to reduce CI times, and one of the options is pruning build jobs as well as test jobs based on what files are changed in the PR. For example, a PR that only modifies the pure Python components of a package should not require rebuilds of either C++ or Cython components. The main blocker to this right now is that the versioning scheme developed above prohibits a mismatched alpha version.
We should be able to fix this by instead building against the most recent version of the dependencies that has a build already finished and uploaded to our nightly channels/indexes. For PRs where no C++ code was changed, the C++ build jobs could be skipped, and the Python jobs could search for the most recent completed C++ build, get the alpha version, and inject that as the build dependency. Then there would still be a tight pinning, avoiding the problems discussed above, but the pinning could be to an older alpha version that is API and ABI compatible, reducing the amount of build duplication needed.
This approach does introduce a few new failure modes that we must consider: