Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Heads-up: Potential Delay in macOS Infrastructure for Node.js v23 #55181

Closed
2 tasks
UlisesGascon opened this issue Sep 30, 2024 · 18 comments
Closed
2 tasks

Heads-up: Potential Delay in macOS Infrastructure for Node.js v23 #55181

UlisesGascon opened this issue Sep 30, 2024 · 18 comments
Assignees
Labels
build-agenda macos Issues and PRs related to the macOS platform / OSX. release-agenda Issues and PRs to discuss during the meetings of the Release team. tsc-agenda Issues and PRs to discuss during the meetings of the TSC. v23.x v23.x Issues that can be reproduced on v23.x or PRs targeting the v23.x-staging branch.

Comments

@UlisesGascon
Copy link
Member

UlisesGascon commented Sep 30, 2024

TL;DR:

We need to confirm it first, but so far, it seems like the macOS infrastructure won't be ready for the Node.js v23 release planned on 2024-10-15 (see Release Issue #1034). This is mainly due to a delay in the Orka migration (see Build Issue #3686).

Original discussion in Slack: Direct Link

Context

How did we arrive at this?

For a long time, the macOS infrastructure has been a challenge for us. Many collaborators and releasers have suffered from it (lack of space, downtimes, etc.). Due to recent events, we ended up having a single provider for the macOS machines, which offered us the opportunity to rethink the infrastructure and mitigate much accumulated technical debt.

After extensive planning for a suitable alternative, we decided to re-engineer our test and release machines using Orka (see Build Issue #3686), thanks to a revamped agreement and a fantastic sponsorship opportunity from MacStadium (read more).

We decided to move from static VMs to ephemeral VMs. This offered us the opportunity to solve all the historical issues we had and allowed us to rebalance the workload more easily. As a result, we are using our resources more efficiently.

What is missing?

In general, we have already finalized the most critical parts of the new infrastructure (new cluster, VPN connections, Orka deployments using Jenkins, image generation using Packer, etc.), but we need more time to finalize the actual testing and migration of the workloads in Jenkins to the new Orka cluster (including the setup of new nodes).

More details: Issue Comment #2278119351

The remaining tasks might take us a bit longer, as we need to wait for more resources to be deployed, and we need to test and patch all the images (macOS version and architecture combinations) in all the pipelines. This is due to the limited availability of the team. I was able to dedicate several weekends in a row to prepare the infrastructure, but currently, I am dealing with some personal things in the following weeks that will limit my availability.

More details: Issue Comment #2352851401

Why does the current infrastructure not support Node.js v23?

Well, Node.js v23 requires a newer version of XCode. This means that we need at least macOS (13.x) in order to support GCC 12 (see Node Pull Request #53668). Our current infrastructure only supports macOS 10.15 and 11.x, which are already obsolete (see). The new infrastructure includes support for macOS 13.x.

What is the impact in terms of the releases?

In practical terms:

  • We can't test macOS as part of the CI process, so we might introduce regressions.
  • We won't be able to generate the signed binaries for macOS. As an example, check Node.js Latest Releases.
    • The files: node-v*-darwin-arm64.tar.gz, node-v*-darwin-arm64.tar.xz, node-v*-darwin-x64.tar.gz, node-v*-darwin-x64.tar.xz, and node-v*.pkg won't be generated. So macOS users won't be able to install it unless they compile it from source.

Next Steps

  • Confirm with @ryanaslett and @UlisesGascon how confident we feel about getting the infrastructure ready on time, so we don't impact the release preparation.
  • @nodejs/build and @nodejs/tsc to coordinate a mitigation plan in case the infrastructure won't be ready.

Potential Options/Mitigations

TBD/TBC

@UlisesGascon UlisesGascon self-assigned this Sep 30, 2024
@richardlau richardlau added tsc-agenda Issues and PRs to discuss during the meetings of the TSC. release-agenda Issues and PRs to discuss during the meetings of the Release team. build-agenda labels Sep 30, 2024
@mhdawson
Copy link
Member

Well, Node.js v23 requires a newer version of macOS (13.x) in order to support GCC 12 (see #53668). Our current infrastructure only supports macOS 10.15 and 11.x, which are already obsolete. The new infrastructure includes support for macOS 13.x.

Is GCC12 already required to build v23? Want to understand if it is an option to build on the existing GCC level initially until we get the new machines. This does add the risk that we'll break people later when we move to the new gcc level, but we can weigh that risk versus not shipping macOS in the initial v23 release as an option.

@richardlau
Copy link
Member

Well, Node.js v23 requires a newer version of macOS (13.x) in order to support GCC 12 (see #53668).

This isn't quite correct as on macOS we use xcode which is clang based (nothing to do with GCC). However there did seem to be general agreement from those more familiar with macOS that we should be on a later version of xcode.

@mhdawson
Copy link
Member

We won't be able to generate the signed binaries for macOS. As an example, check Node.js Latest Releases.

I think @ryanaslett was fixed the release machine, ignoring the version of the compiler used would there be any other blockers for generating the signed binaries?

@richardlau
Copy link
Member

Is GCC12 already required to build v23?

On platforms that use gcc, 12 is the new minimum supported version for Node.js 23. When last checked main still built with gcc 10 but there was at least one PR that was failing to build with gcc 10.

That doesn't apply to macOS as xcode is clang based (see above).

@RedYetiDev RedYetiDev added the macos Issues and PRs related to the macOS platform / OSX. label Sep 30, 2024
@mhdawson
Copy link
Member

mhdawson commented Oct 1, 2024

I think the options that we want to confirm are possible and then choose from include:

  1. Ship 23 on time with macOS based on limited testing in GitHub Actions
  2. Ship 23 on time with macOS based on testing in GitHub actions and a single run on the test CI (not sure if this is possible or not)
  3. Ship 23 on time without macOS
  4. Delay 23 for ??? long

Please add any other options, and identify concerns if some of the options are not possible (for example 2 may not be in the current state).

I don't believe we should consider 4) as I don't think we can guarrantee how long might be ??? and its important to ship on our consistent schedule.

@UlisesGascon
Copy link
Member Author

UlisesGascon commented Oct 1, 2024

Please add any other options, and identify concerns if some of the options are not possible (for example 2 may not be in the current state).

The biggest issue will be signing and uploading the binaries. This is something we always handle using Jenkins and agents, not on our local machines. Not having signed releases would be worse than shipping version 23 without macOS support, as this would break many users and systems. macOS will flag the binaries when you try to install them if they are not properly signed.

@richardlau
Copy link
Member

@UlisesGascon Would you (and/or @ryanaslett) be able to attend the TSC meeting tomorrow (2 Oct) in case there are questions when it is discussed?

@joyeecheung
Copy link
Member

joyeecheung commented Oct 1, 2024

One thing to note is that many developers use macOS anyway, so we should already get good coverage from local testing (or I think if macOS gets broken people would notice very soon). It's unknown what versions of macOS people are using, though.

@mhdawson
Copy link
Member

mhdawson commented Oct 1, 2024

The biggest issue will be signing and uploading the binaries. This is something we always handle using Jenkins and agents, not on our local machines. Not having signed releases would be worse than shipping version 23 without macOS support, as this would break many users and systems. macOS will flag the binaries when you try to install them if they are not properly signed.

I'm not quite understanding the issue here. I think @ryanaslett fixed the existing release machine and I don't think we've stopped doing existing releases so I'm not sure how it will be different for 23 in terms of signing and uploading the binaries?

@UlisesGascon
Copy link
Member Author

@UlisesGascon Would you (and/or @ryanaslett) be able to attend the TSC meeting tomorrow (2 Oct) in case there are questions when it is discussed?

I can make it!

@UlisesGascon
Copy link
Member Author

I'm not quite understanding the issue here. I think @ryanaslett fixed the existing release machine and I don't think we've stopped doing existing releases so I'm not sure how it will be different for 23 in terms of signing and uploading the binaries?

The problem is the MacOS version that we use in the existing release Machine(s) is old (11.x), we will need at least 13.x in order to use a newer version of xcode that can support the release process for Node@23.

AFAIK we need to build the binary in that machine in order to sign it, but maybe this process is different in the Pipelines and we grab the binaries from a different place in order to sign them 🤔

@mhdawson
Copy link
Member

mhdawson commented Oct 1, 2024

@UlisesGascon can you remind me why the release process is different for node 23, assume apple changed something and we have only incorporated the changes into the main/23 line so far?

@targos
Copy link
Member

targos commented Oct 1, 2024

The main question is: would we be fine with releasing v23.0.0 with the current toolchain, and changing it (potentially breaking users on older macOS in a non-major release) later?

@aduh95
Copy link
Contributor

aduh95 commented Oct 1, 2024

2. Ship 23 on time with macOS based on testing in GitHub actions and a single run on the test CI (not sure if this is possible or not)

Would it be possible to disable the Jenkins macOS jobs only on main? In that case, I think we could disable caching (assuming that would solve the disk space issue), meaning macOS builds on Jenkins would take a long time, but that would only affect release lines – and the relatively low frequency at which we need Jenkins builds on release lines, the jobs would be much less likely to build up.

@RedYetiDev RedYetiDev added the v23.x v23.x Issues that can be reproduced on v23.x or PRs targeting the v23.x-staging branch. label Oct 1, 2024
@ryanaslett
Copy link

@UlisesGascon Would you (and/or @ryanaslett) be able to attend the TSC meeting tomorrow (2 Oct) in case there are questions when it is discussed?

I can attend as well

@mhdawson
Copy link
Member

mhdawson commented Oct 2, 2024

Would it be possible to disable the Jenkins macOS jobs only on main? In that case, I think we could disable caching (assuming that would solve the disk space issue), meaning macOS builds on Jenkins would take a long time, but that would only affect release lines – and the relatively low frequency at which we need Jenkins builds on release lines, the jobs would be much less likely to build up.

I think we could do that through the version selector - https://github.com/nodejs/build/blame/main/jenkins/scripts/VersionSelectorScript.groovy although I'm not 100% sure that would catch all CI jobs but it should catch the main PR test jobs.

@richardlau
Copy link
Member

Looks like the V8 13.0 update will require the new infra and won't compile on the existing machines.

Refs: #55014 (comment)

This PR requires newer macOS versions in CI.

@mhdawson
Copy link
Member

I think we can close this out as the release went out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build-agenda macos Issues and PRs related to the macOS platform / OSX. release-agenda Issues and PRs to discuss during the meetings of the Release team. tsc-agenda Issues and PRs to discuss during the meetings of the TSC. v23.x v23.x Issues that can be reproduced on v23.x or PRs targeting the v23.x-staging branch.
Projects
None yet
Development

No branches or pull requests

8 participants