From 07bc06252c926c86501326054079c22015ded229 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Thu, 30 Jan 2025 17:22:10 -0800 Subject: [PATCH 01/13] Add doc on OS onboarding --- docs/project/os-onboarding.md | 52 +++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 docs/project/os-onboarding.md diff --git a/docs/project/os-onboarding.md b/docs/project/os-onboarding.md new file mode 100644 index 00000000000000..4357937b51ffc0 --- /dev/null +++ b/docs/project/os-onboarding.md @@ -0,0 +1,52 @@ +# OS Onboarding Guide + +Adding support for new operating systems (largely just new versions) is a frequent need. This guide describes how we do that, including policies we use. + +References: + +- [.NET OS Support Tracking](https://github.com/dotnet/core/issues/9638) +- [Prereq container image lifecycle](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/lifecycle.md) + +## Context + +In most cases, we find that new OSes _may_ uncover problems in dotnet/runtime and once resolved don't affect up-stack components or apps. This is because nearly all the APIs that touch native code (networking, cryptography) and deal with standard formats (time zones, ASN.1) are in dotnet/runtime. In many cases, we only see test breaks. + +Our testing philosophy is based on risk and past experience. The effective test matrix is huge, the product of OSes \* supported versions \* architectures. We try to make smart choices to skip testing most of the matrix while retaining much of the practical coverage. We also know where we tend to get bitten most when we don't pay sufficient attention. For example, our bug risk across Linux, macOS, and Windows is not uniform. + +## Approach + +New OSes should be added/tested first in `main`. If changes are required, we should prove them out first in `main` before committing to shipping them in a servicing release. However, it isn't always necessary to backport test coverage. + +There are two reasons (beyond known product breaks) to add a new OS reference to a release branch: + +- Add coverage due to practice or known risk +- Update a reference to an EOL OS version + +If those reasons don't apply, then we can often skip backporting new coverage. + +In the case that a .NET version will be EOL in <6 months, then new coverage can typically be skipped. + +## End-of-life + +We will often maintain our level of coverage when a new OS comes available by replacing an older one. This ends up being an effective stratgegy to remediating EOL OSes, ahead of time. + +In some cases, we're required to test an OS version until the end of its life and will need to take specific action to remediate the reference. + +For whatever the reason, we should update references to EOL OSes if we have them. + +## Mechanics + +Most of our testing is done in container images. New images need to be created for each new version in the [dotnet/dotnet-buildtools-prereqs-docker](https://github.com/dotnet/dotnet-buildtools-prereqs-docker) repo. The repo is self-service and largely self-explanatory. One typically creates a new image using the pattern demonstrated by the previous version. + +These images are referenced in our pipeline files: + +- https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/helix-queues-setup.yml +- https://github.com/dotnet/runtime/blob/main/eng/pipelines/libraries/helix.yml + +Those files are for the `main` branch. The same files should be located in the same location in release branches. + +Example PRs: + +- https://github.com/dotnet/runtime/pull/111768 +- https://github.com/dotnet/runtime/pull/111504 +- https://github.com/dotnet/runtime/pull/110492 From 0463ed163b7a033e424de49e0327c7e90cf31049 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Fri, 31 Jan 2025 15:53:20 -0800 Subject: [PATCH 02/13] Update docs/project/os-onboarding.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/project/os-onboarding.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/project/os-onboarding.md b/docs/project/os-onboarding.md index 4357937b51ffc0..8566b95a5f27c3 100644 --- a/docs/project/os-onboarding.md +++ b/docs/project/os-onboarding.md @@ -28,8 +28,7 @@ In the case that a .NET version will be EOL in <6 months, then new coverage can ## End-of-life -We will often maintain our level of coverage when a new OS comes available by replacing an older one. This ends up being an effective stratgegy to remediating EOL OSes, ahead of time. - +We will often maintain our level of coverage when a new OS comes available by replacing an older one. This ends up being an effective strategy to remediating EOL OSes, ahead of time. In some cases, we're required to test an OS version until the end of its life and will need to take specific action to remediate the reference. For whatever the reason, we should update references to EOL OSes if we have them. From f55443c610d3420c1199b477943f17b662fcef30 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Fri, 31 Jan 2025 15:53:36 -0800 Subject: [PATCH 03/13] Update docs/project/os-onboarding.md --- docs/project/os-onboarding.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/project/os-onboarding.md b/docs/project/os-onboarding.md index 8566b95a5f27c3..44825bc15ae305 100644 --- a/docs/project/os-onboarding.md +++ b/docs/project/os-onboarding.md @@ -46,6 +46,8 @@ Those files are for the `main` branch. The same files should be located in the s Example PRs: -- https://github.com/dotnet/runtime/pull/111768 -- https://github.com/dotnet/runtime/pull/111504 -- https://github.com/dotnet/runtime/pull/110492 +- +- +- +- +- From 9f4d7dc63fe720432141b13702b3d1c4c2b9ee0a Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Wed, 5 Feb 2025 13:57:48 -0800 Subject: [PATCH 04/13] Apply suggestions from code review Co-authored-by: Jan Kotas --- docs/project/os-onboarding.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/project/os-onboarding.md b/docs/project/os-onboarding.md index 44825bc15ae305..853aca141302ac 100644 --- a/docs/project/os-onboarding.md +++ b/docs/project/os-onboarding.md @@ -1,4 +1,5 @@ -# OS Onboarding Guide +# Onboarding Guide for Testing of New OS Versions + Adding support for new operating systems (largely just new versions) is a frequent need. This guide describes how we do that, including policies we use. @@ -9,7 +10,8 @@ References: ## Context -In most cases, we find that new OSes _may_ uncover problems in dotnet/runtime and once resolved don't affect up-stack components or apps. This is because nearly all the APIs that touch native code (networking, cryptography) and deal with standard formats (time zones, ASN.1) are in dotnet/runtime. In many cases, we only see test breaks. +In most cases, we find that new OS versions _may_ uncover problems in dotnet/runtime and once resolved don't affect up-stack components or apps. This is because nearly all the APIs that touch native code (networking, cryptography) and deal with standard formats (time zones, ASN.1) are in dotnet/runtime. In many cases, we only see test breaks. + Our testing philosophy is based on risk and past experience. The effective test matrix is huge, the product of OSes \* supported versions \* architectures. We try to make smart choices to skip testing most of the matrix while retaining much of the practical coverage. We also know where we tend to get bitten most when we don't pay sufficient attention. For example, our bug risk across Linux, macOS, and Windows is not uniform. @@ -35,7 +37,8 @@ For whatever the reason, we should update references to EOL OSes if we have them ## Mechanics -Most of our testing is done in container images. New images need to be created for each new version in the [dotnet/dotnet-buildtools-prereqs-docker](https://github.com/dotnet/dotnet-buildtools-prereqs-docker) repo. The repo is self-service and largely self-explanatory. One typically creates a new image using the pattern demonstrated by the previous version. +Linux testing is done in container images. New images need to be created for each new version in the [dotnet/dotnet-buildtools-prereqs-docker](https://github.com/dotnet/dotnet-buildtools-prereqs-docker) repo. The repo is self-service and largely self-explanatory. One typically creates a new image using the pattern demonstrated by the previous version. + These images are referenced in our pipeline files: From 80a1d57300af43f1a18f2be0ac6a59f612c816e4 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Wed, 5 Feb 2025 21:17:41 -0800 Subject: [PATCH 05/13] Expand scope of doc per feedback --- docs/project/os-onboarding.md | 83 +++++++++++++++++++++++++++-------- 1 file changed, 65 insertions(+), 18 deletions(-) diff --git a/docs/project/os-onboarding.md b/docs/project/os-onboarding.md index 853aca141302ac..50fd36b97830de 100644 --- a/docs/project/os-onboarding.md +++ b/docs/project/os-onboarding.md @@ -1,49 +1,88 @@ -# Onboarding Guide for Testing of New OS Versions +# Onboarding Guide for New Operating System Versions +Adding support for new operating systems versions is a frequent need. This guide describes how we do that, including policies we use. -Adding support for new operating systems (largely just new versions) is a frequent need. This guide describes how we do that, including policies we use. +[Porting .NET to a new operating system or architecture](../design/coreclr/botr/guide-for-porting.md) is a related task. Some of these patterns apply, but the overall task will be much larger. References: - [.NET OS Support Tracking](https://github.com/dotnet/core/issues/9638) +- [.NET Support](https://github.com/dotnet/core/blob/main/support.md) - [Prereq container image lifecycle](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/lifecycle.md) -## Context +Internal links: + +- https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/940/Support-for-Linux-Distros +- https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/933/Support-for-Apple-Operating-Systems-(macOS-iOS-and-tvOS) +- https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/939/Support-for-Windows-Operating-Systems -In most cases, we find that new OS versions _may_ uncover problems in dotnet/runtime and once resolved don't affect up-stack components or apps. This is because nearly all the APIs that touch native code (networking, cryptography) and deal with standard formats (time zones, ASN.1) are in dotnet/runtime. In many cases, we only see test breaks. +## Context +In most cases, we find that new OS versions _may_ uncover problems in dotnet/runtime and once resolved don't affect up-stack components or apps. A key design point of our runtime is to be a quite complete cross-platform and -architecture abstraction, so resolving OS compatibility breaks for higher-level code is our enduring intent. -Our testing philosophy is based on risk and past experience. The effective test matrix is huge, the product of OSes \* supported versions \* architectures. We try to make smart choices to skip testing most of the matrix while retaining much of the practical coverage. We also know where we tend to get bitten most when we don't pay sufficient attention. For example, our bug risk across Linux, macOS, and Windows is not uniform. +Nearly all the APIs that touch native code (networking, cryptography) and deal with standard formats (time zones, ASN.1) are in dotnet/runtime. In many cases, we only see test breaks when we onboard a new OS, often from code that tests edge cases. ## Approach -New OSes should be added/tested first in `main`. If changes are required, we should prove them out first in `main` before committing to shipping them in a servicing release. However, it isn't always necessary to backport test coverage. +Our rule is that we declare support for a new OS (for all supported .NET versions) after it is validated in dotnet/runtime `main`. We only hold support on additional testing in special cases (which are uncommon). + +Our testing philosophy is based on percieved risk and past experience. The effective test matrix is huge, the product of OSes \* supported versions \* architectures. We try to make smart choices to skip testing most of the matrix while retaining much of the practical coverage. We also know where we tend to get bitten most when we don't pay sufficient attention. For example, our bug risk across Linux, macOS, and Windows is not uniform. + +We use pragmatism and efficiency to drive our decision making. All things being equal, we'll choose the lowest cost approach. + +## Testing + +Testing is the bread-and-butter of OS onboarding, particularly for a mature runtime like ours. New OS support always needs some form of test enablement. + +Linux and some Windows testing is done in container images. This approach enables us to test many and regularly changing OS versions in a fixed/limited VM environment. The container image creation/update process is self-service. + +We also have VMs (Linux and Windows) and raw metal hardware (Apple) for more direct testing. This is the primary model for Apple and Windows OSes. The VMs are and Apple hardware are relatively slow moving and require [support from dnceng](https://github.com/dotnet/dnceng). + +### Adding coverage + +New OS coverage should be added/tested first in `main`. If changes are required, we should prove them out first in `main` before committing to shipping them in a servicing release, if necessary. -There are two reasons (beyond known product breaks) to add a new OS reference to a release branch: +There are multiple reasons to add a new OS reference to a release branch: -- Add coverage due to practice or known risk -- Update a reference to an EOL OS version +- Known product breaks that require validation and regression testing. +- Past experience suggests that coverage is required to protect against risk. +- OS version is or [will soon go EOL](https://github.com/dotnet/runtime/issues/111818#issuecomment-2613642202) and should be replaced by a newer version. -If those reasons don't apply, then we can often skip backporting new coverage. +For example, we frequently need to backport Alpine updates to release branches to avoid EOL references but less commonly for Ubuntu, given the vast difference in support length. -In the case that a .NET version will be EOL in <6 months, then new coverage can typically be skipped. +A good strategy is to keep `main` at the bleeding edge of new OS versions. That way those references have a decent chance of never needing remediation once they end up in release branches. Being _active_ in `main` enables being _lazy_ in `release/`. -## End-of-life +### Updating or removing coverage -We will often maintain our level of coverage when a new OS comes available by replacing an older one. This ends up being an effective strategy to remediating EOL OSes, ahead of time. -In some cases, we're required to test an OS version until the end of its life and will need to take specific action to remediate the reference. +We will often replace an older OS version with a new one, when it comes available. This approach is an effective strategy of maintaining the same level of coverage and of remediating EOL OSes ahead of time. For the most part, we don't need to care about a specific version. We just want coverage for the OS, like Alpine. -For whatever the reason, we should update references to EOL OSes if we have them. +We should remediate any EOL OS references in our codebase. They don't serve any benefit and come with some risk. -## Mechanics +In the case that a .NET version will be EOL in <6 months, new coverage can typically be skipped. We may even be able to skip remediating EOL OS references. We often opt to stop updating [Supported OSes](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md) late in support period for related reasons. A lazy approach is often the best approach late in the game. Don't upset what's working. -Linux testing is done in container images. New images need to be created for each new version in the [dotnet/dotnet-buildtools-prereqs-docker](https://github.com/dotnet/dotnet-buildtools-prereqs-docker) repo. The repo is self-service and largely self-explanatory. One typically creates a new image using the pattern demonstrated by the previous version. +## Building +Our [build methodology](https://github.com/dotnet/runtime/blob/main/docs/project/linux-build-methodology.md) is oriented around cross-compiling, enabling us to target an old OS version and run on newer ones. It is uncommon for us to need to make changes to the build to address new OS versions, however, there are [rare cases where we need to make adjustments](https://github.com/dotnet/runtime/issues/101944). -These images are referenced in our pipeline files: +We use both containers and VMs for building, depending on the OS. + +Our primary concern is ensuring that we are using [supported operating systems and tools for our build](https://github.com/dotnet/runtime/tree/main/docs/workflow/requirements). + +Our Linux build containers are based on Azure Linux. We [typically need to update them](https://github.com/dotnet/runtime/issues/112191) with a new version of Azure Linux once per release. We do not update the toolset, however. That's fixed, per release. + +## Prereqs containers + +New images need to be created for each new OS version in the [dotnet/dotnet-buildtools-prereqs-docker](https://github.com/dotnet/dotnet-buildtools-prereqs-docker) repo. + +The repo is self-service and largely self-explanatory. One typically creates a new image using the pattern demonstrated by the previous version. Look at commits and [blame](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blame/776324ff16d38e22fd9f06c9842ec338a4b98489/src/alpine/3.20/helix/Dockerfile) to find people who are best suited to help. + +Installing/building the Helix client can be quite involved, particularly for Arm platforms. Don't struggle with that. Just ask for help. + +Test container images are referenced in our pipeline files: - https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/helix-queues-setup.yml - https://github.com/dotnet/runtime/blob/main/eng/pipelines/libraries/helix.yml +- https://github.com/dotnet/runtime/blob/main/eng/pipelines/common/templates/pipeline-with-resources.yml Those files are for the `main` branch. The same files should be located in the same location in release branches. @@ -54,3 +93,11 @@ Example PRs: - - - + +### VMs + +VMs and raw metal environments are used for Linux, macOS, and Windows. They need to be [requested from dnceng](https://github.com/dotnet/dnceng/issues/4307). The turnaround can be long so put in your request before you need it. + +- Linux: All Linux VMs are moving to Azure Linux as a container host. +- macOS: Raw metal hosts for all forms of Apple testing. +- Windows: Windows Client and Server VMs From 0b7d88aa5b528590110c1d7d082c1fd4c044a4d5 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Thu, 6 Feb 2025 07:24:59 -0800 Subject: [PATCH 06/13] Update raw links --- docs/project/os-onboarding.md | 56 ++++++++++++++++++----------------- 1 file changed, 29 insertions(+), 27 deletions(-) diff --git a/docs/project/os-onboarding.md b/docs/project/os-onboarding.md index 50fd36b97830de..5495b46cb60d2d 100644 --- a/docs/project/os-onboarding.md +++ b/docs/project/os-onboarding.md @@ -2,29 +2,26 @@ Adding support for new operating systems versions is a frequent need. This guide describes how we do that, including policies we use. -[Porting .NET to a new operating system or architecture](../design/coreclr/botr/guide-for-porting.md) is a related task. Some of these patterns apply, but the overall task will be much larger. +[Porting .NET to a new operating system or architecture](../design/coreclr/botr/guide-for-porting.md) is a related task. The following patterns likely apply, but the overall task is much larger in scope. References: - [.NET OS Support Tracking](https://github.com/dotnet/core/issues/9638) - [.NET Support](https://github.com/dotnet/core/blob/main/support.md) - [Prereq container image lifecycle](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/lifecycle.md) - -Internal links: - -- https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/940/Support-for-Linux-Distros -- https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/933/Support-for-Apple-Operating-Systems-(macOS-iOS-and-tvOS) -- https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/939/Support-for-Windows-Operating-Systems +- [Support for Linux Distros](https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/940/Support-for-Linux-Distros) (MS internal) +- [Support for Apple Operating Systems](https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/933/Support-for-Apple-Operating-Systems-(macOS-iOS-and-tvOS)) (MS internal) +- [Support for Windows Operating Systems](https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/939/Support-for-Windows-Operating-Systems) (MS internal) ## Context -In most cases, we find that new OS versions _may_ uncover problems in dotnet/runtime and once resolved don't affect up-stack components or apps. A key design point of our runtime is to be a quite complete cross-platform and -architecture abstraction, so resolving OS compatibility breaks for higher-level code is our enduring intent. +In most cases, we find that new OS versions _may_ uncover problems in dotnet/runtime, but don't affect up-stack components or apps once resolved. A key design point of our runtime is to be a quite complete cross-platform and -architecture abstraction, so resolving OS compatibility breaks for higher-level code is an enduring intent. Nearly all the APIs that touch native code (networking, cryptography) and deal with standard formats (time zones, ASN.1) are in dotnet/runtime. In many cases, we only see test breaks when we onboard a new OS, often from code that tests edge cases. ## Approach -Our rule is that we declare support for a new OS (for all supported .NET versions) after it is validated in dotnet/runtime `main`. We only hold support on additional testing in special cases (which are uncommon). +Our rule is that we declare support (for all [supported .NET releases](https://github.com/dotnet/core/blob/main/releases.md)) for a new OS version after it is validated in dotnet/runtime `main`. We will only hold support on additional testing in special cases (which are uncommon). Our testing philosophy is based on percieved risk and past experience. The effective test matrix is huge, the product of OSes \* supported versions \* architectures. We try to make smart choices to skip testing most of the matrix while retaining much of the practical coverage. We also know where we tend to get bitten most when we don't pay sufficient attention. For example, our bug risk across Linux, macOS, and Windows is not uniform. @@ -32,11 +29,11 @@ We use pragmatism and efficiency to drive our decision making. All things being ## Testing -Testing is the bread-and-butter of OS onboarding, particularly for a mature runtime like ours. New OS support always needs some form of test enablement. +Testing is the bread and butter of OS onboarding, particularly for a mature runtime like ours. New OS support always needs some form of test enablement. -Linux and some Windows testing is done in container images. This approach enables us to test many and regularly changing OS versions in a fixed/limited VM environment. The container image creation/update process is self-service. +Linux, Wasm, and some Windows testing is done in container images. This approach enables us to test many and regularly changing OS versions in a fixed/limited VM environment. The container image creation/update process is self-service (discussed later). -We also have VMs (Linux and Windows) and raw metal hardware (Apple) for more direct testing. This is the primary model for Apple and Windows OSes. The VMs are and Apple hardware are relatively slow moving and require [support from dnceng](https://github.com/dotnet/dnceng). +We use VMs (Linux and Windows) and raw metal hardware (Apple) in cases where containers are not practical or for more direct testing. This is the primary model for Apple and Windows OSes. The VMs and Apple hardware are relatively slow to change and require support from dnceng (discussed later). ### Adding coverage @@ -44,7 +41,7 @@ New OS coverage should be added/tested first in `main`. If changes are required, There are multiple reasons to add a new OS reference to a release branch: -- Known product breaks that require validation and regression testing. +- Known product (as opposed to test) breaks that require validation and regression testing. - Past experience suggests that coverage is required to protect against risk. - OS version is or [will soon go EOL](https://github.com/dotnet/runtime/issues/111818#issuecomment-2613642202) and should be replaced by a newer version. @@ -58,41 +55,46 @@ We will often replace an older OS version with a new one, when it comes availabl We should remediate any EOL OS references in our codebase. They don't serve any benefit and come with some risk. -In the case that a .NET version will be EOL in <6 months, new coverage can typically be skipped. We may even be able to skip remediating EOL OS references. We often opt to stop updating [Supported OSes](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md) late in support period for related reasons. A lazy approach is often the best approach late in the game. Don't upset what's working. +In the case that a .NET version will be EOL in <6 (and certainly <3) months, new coverage can typically be skipped. We may even be able to skip remediating EOL OS references. We often opt to stop updating [Supported OSes](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md) late in support period for related reasons. A lazy approach is often the best approach late in the game. Don't upset what's working. ## Building Our [build methodology](https://github.com/dotnet/runtime/blob/main/docs/project/linux-build-methodology.md) is oriented around cross-compiling, enabling us to target an old OS version and run on newer ones. It is uncommon for us to need to make changes to the build to address new OS versions, however, there are [rare cases where we need to make adjustments](https://github.com/dotnet/runtime/issues/101944). -We use both containers and VMs for building, depending on the OS. +We use both containers and VMs for building, depending on the OS. If we test in a container, we likely build in a container. Same for VMs. Our primary concern is ensuring that we are using [supported operating systems and tools for our build](https://github.com/dotnet/runtime/tree/main/docs/workflow/requirements). -Our Linux build containers are based on Azure Linux. We [typically need to update them](https://github.com/dotnet/runtime/issues/112191) with a new version of Azure Linux once per release. We do not update the toolset, however. That's fixed, per release. +Our Linux build containers are based on Azure Linux. We [typically need to update them](https://github.com/dotnet/runtime/issues/112191) with a new version of Azure Linux once per release. We do not update the toolset, however. That's fixed, per release. + +For Apple, we likely need to make an adjustment at each macOS or iOS release to account for an XCode version no longer being supported. ## Prereqs containers -New images need to be created for each new OS version in the [dotnet/dotnet-buildtools-prereqs-docker](https://github.com/dotnet/dotnet-buildtools-prereqs-docker) repo. +New images need to be created for each new OS version in the [dotnet/dotnet-buildtools-prereqs-docker](https://github.com/dotnet/dotnet-buildtools-prereqs-docker) repo, for testing. We also need to create and update images for Linux and Wasm build environments. The repo is self-service and largely self-explanatory. One typically creates a new image using the pattern demonstrated by the previous version. Look at commits and [blame](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blame/776324ff16d38e22fd9f06c9842ec338a4b98489/src/alpine/3.20/helix/Dockerfile) to find people who are best suited to help. Installing/building the Helix client can be quite involved, particularly for Arm platforms. Don't struggle with that. Just ask for help. -Test container images are referenced in our pipeline files: +Container images are referenced in our pipeline files: + +- [eng/pipelines/coreclr/templates/helix-queues-setup.yml](https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/helix-queues-setup.yml) +- [eng/pipelines/libraries/helix.yml](https://github.com/dotnet/runtime/blob/main/eng/pipelines/libraries/helix.yml) +- [eng/pipelines/common/templates/pipeline-with-resources.yml](https://github.com/dotnet/runtime/blob/main/eng/pipelines/common/templates/pipeline-with-resources.yml) -- https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/helix-queues-setup.yml -- https://github.com/dotnet/runtime/blob/main/eng/pipelines/libraries/helix.yml -- https://github.com/dotnet/runtime/blob/main/eng/pipelines/common/templates/pipeline-with-resources.yml +Notes: -Those files are for the `main` branch. The same files should be located in the same location in release branches. +- The first two links are for testing and the last for building. +- The links are for the `main` branch. Release branches should have the same layout. Example PRs: -- -- -- -- -- +- [dotnet/runtime #111768](https://github.com/dotnet/runtime/pull/111768) +- [dotnet/runtime #111504](https://github.com/dotnet/runtime/pull/111504) +- [dotnet/runtime #110492](https://github.com/dotnet/runtime/pull/110492) +- [dotnet/dotnet-buildtools-prereqs-docker #1282](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1282) +- [dotnet/dotnet-buildtools-prereqs-docker #1314](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1314) ### VMs From 19422f53b5682d91b1062d08d7d11e1143fe88a1 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Sat, 8 Feb 2025 10:30:28 -0800 Subject: [PATCH 07/13] Apply suggestions from code review Co-authored-by: Jan Kotas --- docs/project/os-onboarding.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/project/os-onboarding.md b/docs/project/os-onboarding.md index 5495b46cb60d2d..ede49b4e30e4a8 100644 --- a/docs/project/os-onboarding.md +++ b/docs/project/os-onboarding.md @@ -23,7 +23,8 @@ Nearly all the APIs that touch native code (networking, cryptography) and deal w Our rule is that we declare support (for all [supported .NET releases](https://github.com/dotnet/core/blob/main/releases.md)) for a new OS version after it is validated in dotnet/runtime `main`. We will only hold support on additional testing in special cases (which are uncommon). -Our testing philosophy is based on percieved risk and past experience. The effective test matrix is huge, the product of OSes \* supported versions \* architectures. We try to make smart choices to skip testing most of the matrix while retaining much of the practical coverage. We also know where we tend to get bitten most when we don't pay sufficient attention. For example, our bug risk across Linux, macOS, and Windows is not uniform. +Our testing philosophy is based on perceived risk and past experience. The effective test matrix is huge, the product of OSes \* supported versions \* architectures. We try to make smart choices to skip testing most of the matrix while retaining much of the practical coverage. We also know where we tend to get bitten most when we don't pay sufficient attention. For example, our bug risk across Linux, macOS, and Windows is not uniform. + We use pragmatism and efficiency to drive our decision making. All things being equal, we'll choose the lowest cost approach. @@ -33,13 +34,15 @@ Testing is the bread and butter of OS onboarding, particularly for a mature runt Linux, Wasm, and some Windows testing is done in container images. This approach enables us to test many and regularly changing OS versions in a fixed/limited VM environment. The container image creation/update process is self-service (discussed later). -We use VMs (Linux and Windows) and raw metal hardware (Apple) in cases where containers are not practical or for more direct testing. This is the primary model for Apple and Windows OSes. The VMs and Apple hardware are relatively slow to change and require support from dnceng (discussed later). +We use VMs (Linux and Windows) and raw metal hardware (Apple) in cases where containers are not practical or where direct testing is desired. This is the primary model for Apple and Windows OSes. The VMs and Apple hardware are relatively slow to change and require support from dnceng (discussed later). + ### Adding coverage New OS coverage should be added/tested first in `main`. If changes are required, we should prove them out first in `main` before committing to shipping them in a servicing release, if necessary. -There are multiple reasons to add a new OS reference to a release branch: +There are multiple reasons to add a new OS reference in a release branch: + - Known product (as opposed to test) breaks that require validation and regression testing. - Past experience suggests that coverage is required to protect against risk. @@ -55,7 +58,8 @@ We will often replace an older OS version with a new one, when it comes availabl We should remediate any EOL OS references in our codebase. They don't serve any benefit and come with some risk. -In the case that a .NET version will be EOL in <6 (and certainly <3) months, new coverage can typically be skipped. We may even be able to skip remediating EOL OS references. We often opt to stop updating [Supported OSes](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md) late in support period for related reasons. A lazy approach is often the best approach late in the game. Don't upset what's working. +In the case that a .NET version will be EOL in <6 (and certainly <3) months, new coverage can typically be skipped. We may even be able to skip remediating EOL OS references. We often opt to stop updating [supported OSes](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md) late in support period for related reasons. A lazy approach is often the best approach late in the game. Don't upset what's working. + ## Building @@ -67,7 +71,8 @@ Our primary concern is ensuring that we are using [supported operating systems a Our Linux build containers are based on Azure Linux. We [typically need to update them](https://github.com/dotnet/runtime/issues/112191) with a new version of Azure Linux once per release. We do not update the toolset, however. That's fixed, per release. -For Apple, we likely need to make an adjustment at each macOS or iOS release to account for an XCode version no longer being supported. +For Apple, we likely need to make an adjustment at each macOS or iOS release to account for an Xcode version no longer being supported. + ## Prereqs containers From 864f6faa6f671e621a4525e337fe605312846df5 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Sat, 8 Feb 2025 10:52:01 -0800 Subject: [PATCH 08/13] Adding "Environments" as a new section --- docs/project/os-onboarding.md | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/docs/project/os-onboarding.md b/docs/project/os-onboarding.md index ede49b4e30e4a8..fe283e9d0a5f5e 100644 --- a/docs/project/os-onboarding.md +++ b/docs/project/os-onboarding.md @@ -25,7 +25,6 @@ Our rule is that we declare support (for all [supported .NET releases](https://g Our testing philosophy is based on perceived risk and past experience. The effective test matrix is huge, the product of OSes \* supported versions \* architectures. We try to make smart choices to skip testing most of the matrix while retaining much of the practical coverage. We also know where we tend to get bitten most when we don't pay sufficient attention. For example, our bug risk across Linux, macOS, and Windows is not uniform. - We use pragmatism and efficiency to drive our decision making. All things being equal, we'll choose the lowest cost approach. ## Testing @@ -36,7 +35,6 @@ Linux, Wasm, and some Windows testing is done in container images. This approach We use VMs (Linux and Windows) and raw metal hardware (Apple) in cases where containers are not practical or where direct testing is desired. This is the primary model for Apple and Windows OSes. The VMs and Apple hardware are relatively slow to change and require support from dnceng (discussed later). - ### Adding coverage New OS coverage should be added/tested first in `main`. If changes are required, we should prove them out first in `main` before committing to shipping them in a servicing release, if necessary. @@ -60,7 +58,6 @@ We should remediate any EOL OS references in our codebase. They don't serve any In the case that a .NET version will be EOL in <6 (and certainly <3) months, new coverage can typically be skipped. We may even be able to skip remediating EOL OS references. We often opt to stop updating [supported OSes](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md) late in support period for related reasons. A lazy approach is often the best approach late in the game. Don't upset what's working. - ## Building Our [build methodology](https://github.com/dotnet/runtime/blob/main/docs/project/linux-build-methodology.md) is oriented around cross-compiling, enabling us to target an old OS version and run on newer ones. It is uncommon for us to need to make changes to the build to address new OS versions, however, there are [rare cases where we need to make adjustments](https://github.com/dotnet/runtime/issues/101944). @@ -73,10 +70,15 @@ Our Linux build containers are based on Azure Linux. We [typically need to updat For Apple, we likely need to make an adjustment at each macOS or iOS release to account for an Xcode version no longer being supported. +## Environments + +We rely on a set of standard environments for building and testing. These environments are managed with a "config as code" paradigm in our source code. This approach delivers CI reliability -- which we greatly value -- but also comes with the tedious cost of needing to regularly update various reference strings in several files. Updating version strings almost always requires sorting through build breaks, which is a reminder of why we value our approach. + +We may use multiple environments for building and testing a given OS. -## Prereqs containers +### Containers -New images need to be created for each new OS version in the [dotnet/dotnet-buildtools-prereqs-docker](https://github.com/dotnet/dotnet-buildtools-prereqs-docker) repo, for testing. We also need to create and update images for Linux and Wasm build environments. +New container images need to be created for each new OS version in the [dotnet/dotnet-buildtools-prereqs-docker](https://github.com/dotnet/dotnet-buildtools-prereqs-docker) repo. These are used for building and testing Android, Linux, Wasm, and Windows OSes/targets. The repo is self-service and largely self-explanatory. One typically creates a new image using the pattern demonstrated by the previous version. Look at commits and [blame](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blame/776324ff16d38e22fd9f06c9842ec338a4b98489/src/alpine/3.20/helix/Dockerfile) to find people who are best suited to help. @@ -103,8 +105,16 @@ Example PRs: ### VMs -VMs and raw metal environments are used for Linux, macOS, and Windows. They need to be [requested from dnceng](https://github.com/dotnet/dnceng/issues/4307). The turnaround can be long so put in your request before you need it. +VMs and raw metal environments are used for Android, Apple, Linux, and Windows OSes. They need to be [requested from dnceng](https://github.com/dotnet/dnceng/issues/4307). The turnaround can be long so put in your request before you need it. +- Android: Raw metal hosts for some forms of Android testing. +- Apple: Raw metal hosts for all forms of Apple testing. - Linux: All Linux VMs are moving to Azure Linux as a container host. -- macOS: Raw metal hosts for all forms of Apple testing. - Windows: Windows Client and Server VMs + +### Other + +Other environments are typically use a custom process. + +- [Browser Wasm](https://github.com/dotnet/runtime/pull/112066) + From b1342e963758ffe03a0700c9bea218b3e46310c6 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Sat, 8 Feb 2025 10:58:20 -0800 Subject: [PATCH 09/13] Update Xcode casing --- docs/design/datacontracts/Exception.md | 2 +- docs/design/mono/mono-library-mode.md | 2 +- docs/workflow/building/coreclr/ios.md | 2 +- docs/workflow/testing/libraries/testing-apple.md | 4 ++-- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/design/datacontracts/Exception.md b/docs/design/datacontracts/Exception.md index 61455b180d76cb..823e572907e362 100644 --- a/docs/design/datacontracts/Exception.md +++ b/docs/design/datacontracts/Exception.md @@ -50,7 +50,7 @@ ExceptionData GetExceptionData(TargetPointer exceptionAddr) target.ReadPointer(exceptionAddr + /* Exception::StackTraceString offset */), target.ReadPointer(exceptionAddr + /* Exception::RemoteStackTraceString offset */), target.Read(exceptionAddr + /* Exception::HResult offset */), - target.Read(exceptionAddr + /* Exception::XCode offset */), + target.Read(exceptionAddr + /* Exception::Xcode offset */), ); } ``` diff --git a/docs/design/mono/mono-library-mode.md b/docs/design/mono/mono-library-mode.md index f6c6a98e5a746e..b786450c1a0b20 100644 --- a/docs/design/mono/mono-library-mode.md +++ b/docs/design/mono/mono-library-mode.md @@ -109,7 +109,7 @@ public class MainActivity extends AppCompatActivity { After building the mono library with `dotnet publish -r ios-arm64`, it can be found as `lib.dylib` in the binaries folder (i.e. `library-mode-sample/ManagedProject/bin/Release/net8.0/ios-arm64/Bundle/libManagedProject.dylib`). The mono library when built as a shared library with bundling (on by default) can be loaded and used with the following steps: -1. Open/Create the iOS native project in XCode. +1. Open/Create the iOS native project in Xcode. 2. Copy the mono library into the project's root directory (not creating a reference). diff --git a/docs/workflow/building/coreclr/ios.md b/docs/workflow/building/coreclr/ios.md index 9061fd17da438c..9574010e2f8b15 100644 --- a/docs/workflow/building/coreclr/ios.md +++ b/docs/workflow/building/coreclr/ios.md @@ -20,7 +20,7 @@ Build and run the sample app with ./dotnet.sh publish src/mono/sample/iOS/Program.csproj -c Release /p:TargetOS=iossimulator /p:TargetArchitecture=arm64 /p:DeployAndRun=true /p:UseMonoRuntime=false /p:RunAOTCompilation=false /p:MonoForceInterpreter=false ``` -The command also produces an XCode project that can be opened with `open ./src/mono/sample/iOS/bin/iossimulator-arm64/Bundle/HelloiOS/HelloiOS.xcodeproj` and debugged in Xcode. +The command also produces an Xcode project that can be opened with `open ./src/mono/sample/iOS/bin/iossimulator-arm64/Bundle/HelloiOS/HelloiOS.xcodeproj` and debugged in Xcode. ## Running the runtime tests diff --git a/docs/workflow/testing/libraries/testing-apple.md b/docs/workflow/testing/libraries/testing-apple.md index 09e08d1b0cc178..2f6f7128be4d64 100644 --- a/docs/workflow/testing/libraries/testing-apple.md +++ b/docs/workflow/testing/libraries/testing-apple.md @@ -2,10 +2,10 @@ ## Prerequisites -- XCode 11.3 or higher +- Xcode 11.3 or higher - a certificate and provisioning profile if using a device - a simulator with a proper device type and OS version. -Go `XCode > Window > Devices and Simulators` to revise the list of the available simulators and then `"+" button on bottom left > OS Version dropdown selection > Download more simulator runtimes` in case you need to download more simulators. +Go `Xcode > Window > Devices and Simulators` to revise the list of the available simulators and then `"+" button on bottom left > OS Version dropdown selection > Download more simulator runtimes` in case you need to download more simulators. ## Building Libs and Tests From 4df532956c59e21600ed6c3dbc1e579171248cc9 Mon Sep 17 00:00:00 2001 From: Jan Kotas Date: Sat, 8 Feb 2025 11:46:33 -0800 Subject: [PATCH 10/13] Update docs/design/datacontracts/Exception.md --- docs/design/datacontracts/Exception.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/datacontracts/Exception.md b/docs/design/datacontracts/Exception.md index 823e572907e362..61455b180d76cb 100644 --- a/docs/design/datacontracts/Exception.md +++ b/docs/design/datacontracts/Exception.md @@ -50,7 +50,7 @@ ExceptionData GetExceptionData(TargetPointer exceptionAddr) target.ReadPointer(exceptionAddr + /* Exception::StackTraceString offset */), target.ReadPointer(exceptionAddr + /* Exception::RemoteStackTraceString offset */), target.Read(exceptionAddr + /* Exception::HResult offset */), - target.Read(exceptionAddr + /* Exception::Xcode offset */), + target.Read(exceptionAddr + /* Exception::XCode offset */), ); } ``` From ba7e52c39af3c3e2a79ef7b9336f564d28c99c75 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Sun, 9 Feb 2025 14:22:17 -0800 Subject: [PATCH 11/13] Improve intro --- docs/project/os-onboarding.md | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/docs/project/os-onboarding.md b/docs/project/os-onboarding.md index fe283e9d0a5f5e..4dae0a5e4ed0b0 100644 --- a/docs/project/os-onboarding.md +++ b/docs/project/os-onboarding.md @@ -1,8 +1,17 @@ # Onboarding Guide for New Operating System Versions -Adding support for new operating systems versions is a frequent need. This guide describes how we do that, including policies we use. +Adding support for new operating systems versions is a frequent need. This guide describes how we do that, including the policies we use. -[Porting .NET to a new operating system or architecture](../design/coreclr/botr/guide-for-porting.md) is a related task. The following patterns likely apply, but the overall task is much larger in scope. +> Being _active_ in `main` enables being _lazy_ in `release/`. + +This witticism is the underlying philosophy of our approach. By actively maintaining OS versions in our active coding branch, we get the double benefit of bleeding-edge coverage and (in many cases) can avoid EOL remediation cost in `release/` branches. Spending time on avoidable work is a failure of planning. + +> Users are best served when we act _quickly_ not _exhaustively_. + + +This double meaning is instructing us to be boldly pragmatic. Each new OS release brings a certain risk of breakage. The risk is far from uniform across the various repos and components that we maintain. Users are best served when we we've developed 80% confidence and to leave the remaining 20% to bug reports. Exhaustive testing serves no one. + +Continuing with the idea of pragmatism, if you only read this far, you've got the basic idea. The rest of the doc describes more context and mechanics. References: @@ -23,7 +32,9 @@ Nearly all the APIs that touch native code (networking, cryptography) and deal w Our rule is that we declare support (for all [supported .NET releases](https://github.com/dotnet/core/blob/main/releases.md)) for a new OS version after it is validated in dotnet/runtime `main`. We will only hold support on additional testing in special cases (which are uncommon). -Our testing philosophy is based on perceived risk and past experience. The effective test matrix is huge, the product of OSes \* supported versions \* architectures. We try to make smart choices to skip testing most of the matrix while retaining much of the practical coverage. We also know where we tend to get bitten most when we don't pay sufficient attention. For example, our bug risk across Linux, macOS, and Windows is not uniform. +We aim to have "day of" support for about half the OSes we support, including Azure Linux, Ubuntu LTS, and Windows. This means we need to perform ahead-of-time signoff on non-final builds. + +Our testing philosophy is based on perceived risk and past experience. The effective test matrix is huge, the product of OSes \* supported versions \* architectures. We try to make smart choices to **skip testing most of the matrix** while retaining much of the **practical coverage**. We also know where we tend to get bitten most when we don't pay sufficient attention. For example, our bug risk across Linux, macOS, and Windows is not uniform. We use pragmatism and efficiency to drive our decision making. All things being equal, we'll choose the lowest cost approach. @@ -33,7 +44,7 @@ Testing is the bread and butter of OS onboarding, particularly for a mature runt Linux, Wasm, and some Windows testing is done in container images. This approach enables us to test many and regularly changing OS versions in a fixed/limited VM environment. The container image creation/update process is self-service (discussed later). -We use VMs (Linux and Windows) and raw metal hardware (Apple) in cases where containers are not practical or where direct testing is desired. This is the primary model for Apple and Windows OSes. The VMs and Apple hardware are relatively slow to change and require support from dnceng (discussed later). +We use VMs (Linux and Windows) and raw metal hardware (Android and Apple) in cases where containers are not practical or where direct testing is desired. This is the primary model for Apple and Windows OSes. The VMs and mobile/Apple hardware are relatively slow to change and require support from dnceng (discussed later). ### Adding coverage @@ -41,20 +52,19 @@ New OS coverage should be added/tested first in `main`. If changes are required, There are multiple reasons to add a new OS reference in a release branch: - - Known product (as opposed to test) breaks that require validation and regression testing. - Past experience suggests that coverage is required to protect against risk. - OS version is or [will soon go EOL](https://github.com/dotnet/runtime/issues/111818#issuecomment-2613642202) and should be replaced by a newer version. For example, we frequently need to backport Alpine updates to release branches to avoid EOL references but less commonly for Ubuntu, given the vast difference in support length. -A good strategy is to keep `main` at the bleeding edge of new OS versions. That way those references have a decent chance of never needing remediation once they end up in release branches. Being _active_ in `main` enables being _lazy_ in `release/`. +A good strategy is to keep `main` at the bleeding edge of new OS versions. That way those references have a decent chance of never needing remediation once they end up in release branches. ### Updating or removing coverage We will often replace an older OS version with a new one, when it comes available. This approach is an effective strategy of maintaining the same level of coverage and of remediating EOL OSes ahead of time. For the most part, we don't need to care about a specific version. We just want coverage for the OS, like Alpine. -We should remediate any EOL OS references in our codebase. They don't serve any benefit and come with some risk. +We should remediate any EOL OS references in our codebase. They don't serve any benefit and come with some risk. They are also likely to result in compliance tickets (that come with a deadline) that we want to avoid. In the case that a .NET version will be EOL in <6 (and certainly <3) months, new coverage can typically be skipped. We may even be able to skip remediating EOL OS references. We often opt to stop updating [supported OSes](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md) late in support period for related reasons. A lazy approach is often the best approach late in the game. Don't upset what's working. @@ -118,3 +128,6 @@ Other environments are typically use a custom process. - [Browser Wasm](https://github.com/dotnet/runtime/pull/112066) +## Porting + +[Porting .NET to a new operating system or architecture](../design/coreclr/botr/guide-for-porting.md) is a related task. The same patterns likely apply, but the overall task is much larger in scope. From cda2bcf8b21d3141c048e18974f384b9551ce53b Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Sun, 9 Feb 2025 14:25:49 -0800 Subject: [PATCH 12/13] Fix typo --- docs/project/os-onboarding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/project/os-onboarding.md b/docs/project/os-onboarding.md index 4dae0a5e4ed0b0..0746546b10ad8a 100644 --- a/docs/project/os-onboarding.md +++ b/docs/project/os-onboarding.md @@ -9,7 +9,7 @@ This witticism is the underlying philosophy of our approach. By actively maintai > Users are best served when we act _quickly_ not _exhaustively_. -This double meaning is instructing us to be boldly pragmatic. Each new OS release brings a certain risk of breakage. The risk is far from uniform across the various repos and components that we maintain. Users are best served when we we've developed 80% confidence and to leave the remaining 20% to bug reports. Exhaustive testing serves no one. +This double meaning is instructing us to be boldly pragmatic. Each new OS release brings a certain risk of breakage. The risk is far from uniform across the various repos and components that we maintain. Users are best served when we've developed 80% confidence and to leave the remaining (potential) 20% to bug reports. Exhaustive testing serves no one. We've also found that are users do a great job finding corner cases and enthusiastically participate in the process by opening issues in the appropriate repo. Continuing with the idea of pragmatism, if you only read this far, you've got the basic idea. The rest of the doc describes more context and mechanics. From 42634cdd09af78149efbc5ae25f85ed13ec4e265 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Sun, 9 Feb 2025 16:25:34 -0800 Subject: [PATCH 13/13] Update docs/project/os-onboarding.md Co-authored-by: Jan Kotas --- docs/project/os-onboarding.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/project/os-onboarding.md b/docs/project/os-onboarding.md index 0746546b10ad8a..c233c5ebdfab86 100644 --- a/docs/project/os-onboarding.md +++ b/docs/project/os-onboarding.md @@ -9,7 +9,8 @@ This witticism is the underlying philosophy of our approach. By actively maintai > Users are best served when we act _quickly_ not _exhaustively_. -This double meaning is instructing us to be boldly pragmatic. Each new OS release brings a certain risk of breakage. The risk is far from uniform across the various repos and components that we maintain. Users are best served when we've developed 80% confidence and to leave the remaining (potential) 20% to bug reports. Exhaustive testing serves no one. We've also found that are users do a great job finding corner cases and enthusiastically participate in the process by opening issues in the appropriate repo. +This double meaning is instructing us to be boldly pragmatic. Each new OS release brings a certain risk of breakage. The risk is far from uniform across the various repos and components that we maintain. Users are best served when we've developed 80% confidence and to leave the remaining (potential) 20% to bug reports. Exhaustive testing serves no one. We've also found that our users do a great job finding corner cases and enthusiastically participate in the process by opening issues in the appropriate repo. + Continuing with the idea of pragmatism, if you only read this far, you've got the basic idea. The rest of the doc describes more context and mechanics.