DISCUSS: Raise MacOS minimum target from 10.9 to 10.13?

[h-vetinari]

current todo list

• make an announcement (add announcement for dropping MacOS<10.13 #1993)
• decide how we want to enforce the __osx constraint
  • going to use recipe_append.yaml generated by conda-forge-ci-setup (add recipe_append.yaml for osx with __osx >=10.13 in run conda-forge-ci-setup-feedstock#260)
  • The append doesn't work for recipes with outputs, so we'll have to use a run export on the compilers for osx.
• Do META: {{ stdlib("c") }} migration #2102
• Decide if we will allow folks to set a min version less the global minimum.
  • If yes, do nothing. stdlib jinja mechanism will allow this
  • If no, then we need a minimigrator to adjust / remove any custom minimum pins below 10.13
• clean up recipes with outdated pins in CBC (that don't explicitly want/need it)
• figure out if there are any finicky builds on osx that need adjusting? I recall some but maybe that is out of date by now.
• figure out remaining(?) places in the infrastructure that involve assumptions about the sysroot and fix them up
• patch old builds with 10.9 __osx constraint?
• bump c_stdlib_version to 10.13 for osx-64 in global pinnings

issue text from start

When azure deprecated the macOS-10.15 images, it turned out macOS-11 still supports targets all the way back to our baseline 10.9, and so it was decided to separate the discussion of the image upgrade from the default MACOSX_DEPLOYMENT_TARGET.

However our baseline target 10.9 is now EOL since almost 6(!) years, so I guess that discussion should be had at some point. I don't think there's a really urgent need (what with shipping our own libcxx, plus the _LIBCPP_DISABLE_AVAILABILITY mechanism¹), but this topic recently came up in a numpy discussion, so I though I'd open an issue.

Originally, I thought our hand would be forced once macOS-11 images are deprecated, but it turns out that even in the macOS-12 images, there's still an SDK with Xcode 13.1, which in turn still supports targets back to 10.9. Only once we're forced to use Xcode 14+ would we have to bump the minimum target to 10.13.

July 2023 update

Out of curiosity, I wanted to check when the last bump of the MACOSX_DEPLOYMENT_TARGET happened, and, as it turns out, it's been at 10.9 since the initial commit of https://github.com/conda-forge/conda-forge-pinning-feedstock 🤯

With a bit of digging (and luck), I found the bump from 10.7 to 10.9 though: conda-forge/toolchain-feedstock@7a470c5 - 7 years ago. At the time, 10.9 was just before its EOL. If we applied the same standard today we should jump directly to 11.0.

Given that most users these days are on much newer versions, and want to use newer features (relevant for our packaging, like support for metal or the new LAPACK implementation), I think it might make sense to stop dragging our feet so much on this. We're (slowly but steadily) moving with the time on linux and windows as well, why should osx need to fall so far behind?

To substantiate this a bit more, I wanted to look at the usage numbers for different MacOS versions. The broadest measure is "everyone who uses a browser", but there are apparently no good usage numbers, because Apple keeps misreporting its OS version in HTTP headers, for some complicated reasons. I did find however that 92% are on 10.15+, which is the version every version after that pretends to be. Notably, all (distinguishable) versions that're EOL have max 1-1.5% usage numbers (<7% cumulated) -- and again, this is all MacOS users, not just those of conda-forge.

Footnotes

1. though there are quite a few feedstocks that just bump the MACOSX_DEPLOYMENT_TARGET because it happens to unbreak CI - not least because the compiler errors point in this direction ("X is unavailable: introduced in macOS Y.Z") - but often not adding __osx >={{ MACOSX_DEPLOYMENT_TARGET|default("10.9") }} as a dependence; such packages might already be broken on old OSX anyway... ↩

[isuruf]

FYI: We are not restricted by the xcode version and our builds are completely independent of the macos build image.
See https://github.com/conda-forge/conda-forge-ci-setup-feedstock/blob/main/recipe/download_osx_sdk.sh

[ngam]

The phracker repo we get the SDKs from is no longer maintained.

our builds are completely independent of the macos build image

Yes, but we also use a hack to expose recent Metal stuff, e.g. in pytorch:

• https://github.com/conda-forge/pytorch-cpu-feedstock/blob/5df993a140f950439191ac1149045b7239eddf68/conda-forge.yml#L1-L5
• https://github.com/conda-forge/pytorch-cpu-feedstock/blob/5df993a140f950439191ac1149045b7239eddf68/recipe/conda_build_config.yaml#L1-L2

Maybe irrelevant to the main discussion here though...

[h-vetinari]

One aspect of this discussion that's flown under the radar for a long time is now starting to come to the fore. While we can patch around the C++ standard library by shipping our own up-to-date libcxx, we're dependent on the sdk for an up-to-date C standard library.

So far, many projects have not relied much on newer C functionality, but certain projects (crucially LLVM, which is our main compiler on osx) now starts to rely on C11 features more and more. I've had a lot of problems with building libcxx 15, and one of the big issues was the lack of C11 support on osx (and linux).

I've now solved this by reverting a commit that removed a lot of workarounds in the LLVM 15 timeframe, but this is not a long-term solution. Of course, we could raise the MACOSX_DEPLOYMENT_TARGET for libcxx, but given its central role in conda-forge, that's likely effectively equivalent to bumping the minimum version everywhere.

I haven't yet managed to google (resp. experiment) which SDK added the required C11 functions. I can do that if people are interested.

[h-vetinari]

libcxx 17 has recently dropped support for macOS<10.13 when building the shared libcxx library (I noted this in conda-forge/libcxx-feedstock#110 when the RFC opened).

As far as I can tell, this is more fundamental than the availability workarounds we do (for building projects on top of libcxx), as it will really become impossible to build (shared) libcxx itself for targets <10.13.

Like for other fundamental toolchain dependencies (like upstream requiring vs2019 vs. vs2017 recently), I don't think it's feasible to try to keep patching back in support for older macos versions, as there will be zero coverage upstream and therefore things will bitrot fast.

Seeing that LLVM is our default toolchain on osx (and assuming I'm not overlooking something), I think we'll have to increase the minimum version to 10.13 at the latest when we switch our default compilers on osx to clang 17 (earliest that could happen would be late 2023; of course we could drag it out for quite a while longer by staying on clang 16 for another year or so afterwards).

PS. It's worth noting that libcxx had been waiting on chrome to bump their required minimum version to 10.13, which was apparently the last major "holdout" (from the POV of LLVM). So I think end of 2023 would actually a very reasonable timeline to bump to 10.13; at that point 10.12 will have been EOL for 4 years.

[h-vetinari]

@h-vetinari: It's worth noting that libcxx had been waiting on chrome to bump their required minimum version to 10.13, which was apparently the last major "holdout" (from the POV of LLVM)

I just noticed that abseil (and google more broadly¹) also raised its minimum to 10.13, and uses as official policy what LLVM did informally, i.e. MacOS support is defined in this document as:

We will support [macOS] back to the oldest macOS target platform needed by Chrome

There's also an overview in this table.

Reminder: To the degree that the lower bound of the SDK depends on C++'s stdlib, we can continue to circumvent this with _LIBCPP_DISABLE_AVAILABILITY, but on the C side there are no workarounds. From the outside, it's hard to say if a project is relying on C's stdlib functionality, but use of C11's aligned_alloc (including std::aligned_alloc on the C++ side) is a sure sign, for example.

Footnotes

1. notably protobuf, but also google-cloud-cpp, gtest, and probably several others ↩

[mfansler]

Just want to chime in that several feedstocks are requiring the -D_LIBCPP_DISABLE_AVAILABILITY flag to get passing on osx-64 for the r-base=4.3 migration builds. This is specific to R 4.3 builds, which default to C++17 standard.

It's not widespread, but it is a front where the continued support for 10.9 adds to maintainer workload.

[potiuk]

I am not 100% sure if that's the case but I believe if nothing is done, there will be a flurry of problems Conda maintainers will have to deal with.

If nothing is done about the issue, Conda users will start having a lot more problems soon, so I suggest Conda maintainers to make some bold moves there.

One of the problems that conda users are going to experience is more and more packages not supporting binary version of packages for 10.9 will fail compilation. This is already happening with Apache airflow installing google-re2 package. If someone has a conda installation done ~ 10.9 (I guess), they cannot install Airflow because conda tries to compile a 10.9 variant of the packag as it has no binary version and it fails.

Apparently reinstalling conda from the scratch helps. This is what we recommend our users for now.

We do not officially support conda as installation medium, and if this is will not be fixed soon and we get more issues, we might want to officially say it does not work with conda and direct them to only use pip if this is not handled somehow - we are at a loss because we cannot do anything with that, other than telling our users to reinstall conda or move to pip.

I would really appreciate if conda maintainers have a solution for that, we are going to redirect our users to this issue here like we did with apache/airflow#32852 (reply in thread)

@xylar - I believe you were at some point in time maintainer of Airlfow packages on conda, I would really appreciate if this problem is taken off our heads at Airflow and handled by Conda maintainers.

[h-vetinari]

If someone has a conda installation done ~ 10.9 (I guess), they cannot install Airflow because conda tries to compile a 10.9 variant of the packag as it has no binary version and it fails.

I'm not sure if you're confusing conda and pip here, but conda will not compile anything when a package is not found (this is the default behaviour of pip though), which seems to be consistent with apache/airflow#32852.

It's also worth noting that this is in some ways the opposite problem than what this issue is about (aside from the fact that cross-use of conda & pip is always tricky, not recommended and not supported), namely that someone who has a new MacOS (13.5) is being wrongly detected as an old one. This issue is about raising the very conservative baseline version (10.9) against which our packages are compiled (successfully).

I have a vague suspicion that what's happening is that someone has a conda environment, tries to pip-install a package, pip doesn't find a wheel for some reason and thus compiles from source, the compilation within the environment picks up and old MacOS SDK. If this issue is solved by reinstalling (==updating) conda, then this might also just be an old conda bug.

In any case, I'd appreciate if you didn't start casting blame before understanding the mechanics of the situation more deeply.

[potiuk]

Unfortunately I do not know how conda works nor have a history of conda under my belt and I do not think I should learn old conda bugs to help out usera who hit it.

As I wrote, this was the suspicion I had but I clearly stated that I do not know. I just know that conda users might get into the situation, and are blaming airflow. So I am not blaming specific problem just trying to guess. If it is an old conda bug, it's still a conda problem not airflow :). And it's not blaming anyone is just directing the users to the place where helpful conda maintainers might help our users or maybe even are made aware about the issue and fix it permanently.

Just wanted to raise awareness of it because at the beginning of the issue or was written it is not a widespread issue - and I am signalling thar it might become one soon. And I guess conda maintainers are the only ones that can diagnose it and help (also their users - because this is as much conda as Airflow users we are talking her about).

So yeah. I am not blaming, I am trying to direct the user problem to the place where the users can get help. If conda maintainers will help somehow by fixing conda, cool. If this is a conda bug, cool as well - same place, same people.

In either case we will keep redirecting similar problems here (after proposing workaround with reinstallation) because we can't do anything else about it

I hope it is clear and no bad feelings.