First Version of E2E Validation Infrastructure

[Elliot (theelliotm)]

📖 Description

This PR introduces an E2E validation testing matrix that uses self-hosted runners to run tests across multiple OS versions. Currently includes support for:

x64

• Windows: Prerelease, 25H2, 24H2, 23H2
• MacOS: 26, 15
• Linux: Ubuntu 26.04, Ubuntu 24.04, Debian 13, RHEL 10

arm64
As of now, there are no Azure VMs that support nested virt on ARM CPUs.

Current Backend Support

• Process-T1: ✅ GOOD. Supported on prerelease Windows. Test failures are either a fault of the testing suite or legitimate issues with MXC.
• Process-T3: ✅ GOOD. The test suite currently can't be manually run with T1 or T3, so testing T3 on T1 systems doesn't work. Other test failures are either a fault of the testing suite or legitimate issues with MXC.
• Sandbox: ⛔ NOT IMPLEMENTED
• WSLC: ⚠️✅ (MOSTLY) GOOD. There is a chance of hitting a rate-limit when updating WSL and downloading WSLC images in the test script itself. Other test failures are either a fault of the testing suite or legitimate issues with MXC. Future fix: Split WSLC into multiple jobs spaced 15 minutes apart.
• Isolation Session: ⚠️ ISSUE: Feature_AgentSessionsBaseSupport is not enabled on the VM. Should be enabled by default soon!
• Bubblewrap: ✅ GOOD.
• LXC: ⚠️✅ (MOSTLY) GOOD: Some networking tests fail on versions besides Ubuntu 24. I'm not sure if it's the VM's fault or an issue with MXC.
• MicroVM: ⛔ NOT WORKING: (Windows) Both cold and warm starts freeze and timeout. (Linux) No E2E test suite available.
• Hyperlight: ⛔ NOT IMPLEMENTED
• Seatbelt: ✅ GOOD. Currently no official E2E test suite, but a temporary one I had Copilot write didn't have any issues running or using Seatbelt APIs.

Schedule

The full test matrix runs nightly on main. Once we get bugs fixed and a clearer idea of cost, the schedule will be adjusted.

The testing matrixes and all self-hosted runners are fully configurable in /scripts/ci/validation-test-matrix.json.
The schedule timing is configurable in /.github/workflows/Validation.Tests.Scheduled.yml.

🔗 References

🔍 Validation

No MXC code was modified. All newly created workflows can run and provide feedback on resolving issues.

✅ Checklist

• Signed the Contributor License Agreement
• Updated documentation (if applicable)
• Updated Copilot instructions (if build, architecture, or conventions changed)

📋 Issue Type

• Feature

Microsoft Reviewers: Open in CodeFlow

[Copilot]

Pull request overview

Copilot reviewed 153 out of 271 changed files in this pull request and generated no new comments.

Suppressed comments (1)

scripts/ci/validation-test-matrix.json:330

• The PR description says the full matrix runs nightly and lists macOS 26/15 with Seatbelt support, but this plan contains no macOS trigger. Since the Seatbelt dispatcher also exits 2, the resolver emits no macOS validation jobs at all. Either wire the Seatbelt suite and add these platforms to nightly, or update the PR description to state that macOS entries are declared but dormant.

    "nightly": [

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

scripts/ci/resolve-validation-test-matrix.mjs:65

• A missing pool currently passes validation because only non-null values are type-checked, but expandPlan then silently drops that architecture at line 149. An explicit empty string is the documented dormant state; omitting the field is malformed and likely a catalog typo. Require every non-macOS architecture to declare a string pool (which may still be empty).

      } else if (details.pool != null && typeof details.pool !== 'string') {
        throw new Error(`${platform.id}.${architecture}.pool must be a string`);

.github/workflows/Validation.Tests.Scheduled.yml:6

• The PR description says the schedule is fully configurable in validation-test-matrix.json, but the cron expressions and dispatch choices are hardcoded in this workflow (and docs/ci-validation-infrastructure.md:285-290 correctly says schedule changes belong here). Please update the PR description so it does not direct maintainers to the catalog for schedule changes.

  schedule:
    - cron: '0 8 * * 1-6'
    - cron: '0 8 * * 0'

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.

Suppressed comments (3)

sdk/node/tests/integration/test-helpers.ts:52

• These new required package binaries are only added to the GitHub artifact manifest. The Azure build still copies Windows files exclusively through .azure-pipelines/templates/Rust.Build.Job.yml's signPattern, which omits both executables, while its SDK integration job runs this same package test on Windows. Official Azure builds will therefore fail with both files missing. Add them to the Azure signing/copy manifest as well, or keep validation-only binaries out of the SDK package expectations consistently.

  'wxc-ui-probe.exe',
  'wxc-test-driver.exe',

scripts/ci/resolve-validation-test-matrix.mjs:25

• The new catalog validator and expansion logic has no automated tests, despite controlling every scheduled combination and containing important branches for malformed catalogs, dormant pools, architecture suppression, and empty family matrices. Similar JavaScript infrastructure under scripts/versioning/tests/ is unit-tested. Add Node tests for validateCatalog, expandPlan, and resolvePlan, and run them in CI so matrix regressions are caught before a scheduled workflow consumes them.

export function validateCatalog(catalog) {

.github/workflows/Validation.Tests.Scheduled.yml:6

• The PR description says the schedule is fully configurable in validation-test-matrix.json, but the actual timing is hard-coded here (and the repository documentation correctly says schedule changes belong in this workflow). Please update the PR description to distinguish the cron schedule from the catalog-controlled plans, pools, and backend combinations.

    - cron: '0 8 * * 1-6'
    - cron: '0 8 * * 0'

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

sdk/node/tests/integration/test-helpers.ts:52

• These new required package binaries are only added to the GitHub artifact manifest. The official Azure build still copies its Windows artifact from signPattern in .azure-pipelines/templates/Rust.Build.Job.yml:49-58,185-191, which omits both files, while Package.NpmSdk.Job.yml:51-57 packages that whole artifact. GitHub and official Azure npm tarballs will therefore have different payloads. Add both executables to the Azure build/signing manifest as part of this change (or keep them out of the SDK expectation).

  'wxc-ui-probe.exe',
  'wxc-test-driver.exe',

tests/scripts/run_ci_backend_tests.sh:51

• MXC_LXC_TESTS_REQUIRE_EXECUTION is not read anywhere by run_lxc_all_tests.sh (the only repository occurrence is this assignment). That suite counts exit-77 cases as skips, and even its all-skipped path only warns before exiting 0 (run_lxc_all_tests.sh:73-85), so an LXC matrix job can still be green without the required coverage. Implement the flag in the aggregate suite (and fail on the intended skip condition), or have this dispatcher inspect the summary and return nonzero.

        MXC_LXC_TESTS_REQUIRE_EXECUTION=1 bash "$script_root/run_lxc_all_tests.sh"

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

scripts/ci/resolve-validation-test-matrix.mjs:65

• A missing pool is accepted here even though it is required for Windows/Linux. expandPlan then treats the missing value like a dormant empty pool and silently emits no job, so a typo in this field can make requested validation disappear while the resolver succeeds. Require the field to be a string; keep "" as the explicit dormant value.

      if (platform.family === 'macos') {
        assertNonEmptyString(details.runner, `${platform.id}.${architecture}.runner`);
      } else if (details.pool != null && typeof details.pool !== 'string') {
        throw new Error(`${platform.id}.${architecture}.pool must be a string`);
      }

scripts/ci/prepare-linux-host.sh:177

• With set -euo pipefail, this unguarded assignment exits host preparation when ip fails (notably after start_lxc_bridge deliberately warns and returns because the bridge is absent). That contradicts this function's diagnostic-only contract and prevents the backend suite from reporting the actual failure. Make the probe tolerate an absent bridge.

    local bridge_ip
    bridge_ip="$(ip -4 -o addr show "$bridge" 2>/dev/null |
        awk '{print $4}' | cut -d/ -f1 | head -n 1)"

.github/workflows/Build.Windows.Job.yml:119

• These test-only executables are now part of the GitHub product artifact, and Package.NpmSdk.Job.yml:22-49 copies that whole artifact into sdk/node/bin, so GitHub-built npm packages ship internal harnesses that consumers do not need. The Azure artifact excludes them via .azure-pipelines/templates/Rust.Build.Job.yml:49-58, making package contents producer-dependent. Publish a separate validation artifact or explicitly strip these files before npm pack.

            src/target/${{ matrix.target }}/release/wxc-ui-probe.exe
            src/target/${{ matrix.target }}/release/wxc-test-driver.exe

scripts/ci/resolve-validation-test-matrix.mjs:25

• This scheduling-critical resolver has no automated tests; resolving the checked-in nightly catalog only exercises one happy path. Add Node tests for malformed catalogs, empty versus missing pools, ARM64 suppression, duplicate requests, and per-family output, following the existing scripts/versioning/tests pattern.

export function validateCatalog(catalog) {

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

scripts/ci/prepare-linux-host.sh:177

• This diagnostic can terminate host preparation when the bridge is absent. With set -euo pipefail, the failed ip ... show makes the assignment return nonzero, despite start_lxc_bridge deliberately treating an absent bridge as a warning and this function promising never to fail. Neutralize the lookup failure so the backend suite, rather than diagnostics, determines the result.

    local bridge_ip
    bridge_ip="$(ip -4 -o addr show "$bridge" 2>/dev/null |
        awk '{print $4}' | cut -d/ -f1 | head -n 1)"

scripts/ci/resolve-validation-test-matrix.mjs:25

• This new resolver is the sole gate for catalog validity and scheduling, but no automated tests cover its validation, architecture suppression, family splitting, or empty-plan behavior. A regression can silently remove E2E coverage or allocate the wrong specialized runners. Add Node tests for valid expansion plus malformed/missing fields and dormant pools.

export function validateCatalog(catalog) {