feat: GitHub-hosted runners for benchmarks (#29955)

## **Description** ### Major changes - **benchmarks.yml** - Does some important and new-to-us things like: - ~~Self-hosted runner `runs-on: gha-mm-scale-set-ubuntu-22.04-amd64-med`~~ - ~~Running in the same kind of Docker container we use in CircleCI `container: image: cimg/node:22.13-browsers`~~ - ~~Unlike on CircleCI, does not work without installing `Xvfb`~~ - **codespaces.yml** - I originally added this workflow, and it was important to caching in the past, but with the new workflows in GHA, it's no longer needed, so I deleted it - **test-circular-deps.yml** - We were hitting the workflow limit of 20, so I inlined this simple workflow, that did not need to be reusable ### Prerequisites to merging this PR - ✔️ Merge this `benchmarks` branch of `github-tools`: https://github.com/MetaMask/github-tools/blob/benchmarks/.github/actions/setup-environment/action.yml - ✔️ Change `setup-environment@benchmarks` to `setup-environment@1d657e262aea7e3f216754febb624831527d2565` - ~~Vastly increase the number of allocated self-hosted runners~~ (no longer needed because not using self-hosted anymore) ### This is just Part 1 of a larger 4-part task to make the startup time quality gate, but I think it's the hardest part 1. Run on GitHub-hosted runners so that hopefully the numbers will be more stable (this PR) 2. Get this to measure the same measurements in the same way as we do for Sentry 3. Make metamaskbot post with the results of this instead of the current CircleCI source 4. Set up the quality gate [![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/MetaMask/metamask-extension/pull/29955?quickstart=1) ## **Related issues** Progresses: MetaMask/MetaMask-planning#3679  --------- Co-authored-by: Norbert Elter <[email protected]>
MetaMask · Feb 18, 2025 · 2b39e3b · 2b39e3b
1 parent e758c07
commit 2b39e3b
Show file tree

Hide file tree

Showing 29 changed files with 133 additions and 74 deletions.
diff --git a/.github/workflows/add-release-label.yml b/.github/workflows/add-release-label.yml
@@ -18,7 +18,7 @@ jobs:
           fetch-depth: 0 # This is needed to checkout all branches
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Get the next semver version
         id: get-next-semver-version

diff --git a/.github/workflows/build-beta.yml b/.github/workflows/build-beta.yml
@@ -42,7 +42,7 @@ jobs:
 
       - name: Setup environment
         if: ${{ steps.needs-beta-build.outputs.NEEDS_BETA_BUILD == 'true' }}
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Run beta build
         if: ${{ steps.needs-beta-build.outputs.NEEDS_BETA_BUILD == 'true' }}

diff --git a/.github/workflows/build-storybook.yml b/.github/workflows/build-storybook.yml
@@ -15,7 +15,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Build storybook
         run: yarn storybook:build

diff --git a/.github/workflows/check-attributions.yml b/.github/workflows/check-attributions.yml
@@ -12,7 +12,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Check attributions changes
         run: yarn attributions:check
diff --git a/.github/workflows/check-pr-labels.yml b/.github/workflows/check-pr-labels.yml
@@ -21,7 +21,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Check PR has required labels
         env:

diff --git a/.github/workflows/check-template-and-add-labels.yml b/.github/workflows/check-template-and-add-labels.yml
@@ -14,7 +14,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Check template and add labels
         id: check-template-and-add-labels

diff --git a/.github/workflows/close-bug-report.yml b/.github/workflows/close-bug-report.yml
@@ -16,7 +16,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Close release bug report issue
         env:

diff --git a/.github/workflows/codespaces.yml b/.github/workflows/codespaces.yml
diff --git a/.github/workflows/fitness-functions.yml b/.github/workflows/fitness-functions.yml
@@ -17,7 +17,7 @@ jobs:
           fetch-depth: 0 # This is needed to checkout all branches
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Run fitness functions
         env:

diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
@@ -6,6 +6,7 @@ on:
       - main
       - master
       - Version-v*
+      - trigger-ci*
   pull_request:
     types:
       - opened
@@ -16,7 +17,7 @@ on:
 jobs:
   lint-workflows:
     name: Lint workflows
-    uses: metamask/github-tools/.github/workflows/lint-workflows.yml@c534f265e02af2f2422a3c686bb09a11bfbf4cc2
+    uses: metamask/github-tools/.github/workflows/lint-workflows.yml@1d657e262aea7e3f216754febb624831527d2565
 
   test-lint-shellcheck:
     name: Test lint shellcheck
@@ -28,7 +29,16 @@ jobs:
 
   test-circular-deps:
     name: Test circular deps
-    uses: ./.github/workflows/test-circular-deps.yml
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Setup environment
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
+
+      - name: Check circular dependencies
+        run: yarn circular-deps:check
 
   test-lint-changelog:
     name: Test lint changelog
@@ -66,10 +76,52 @@ jobs:
     name: Validate lavamoat policy webapp
     uses: ./.github/workflows/validate-lavamoat-policy-webapp.yml
 
+  prep-build-test-browserify:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Setup environment
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
+
+      - run: yarn build:test
+
+      - name: Upload artifact prep-build-test-browserify
+        uses: actions/upload-artifact@v4
+        with:
+          name: prep-build-test-browserify
+          path: ./dist/
+
+  prep-build-test-webpack:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Setup environment
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
+
+      - run: yarn webpack --test --no-lavamoat --no-cache --browser=chrome --lockdown --sentry --snow --env production
+        env:
+          SEGMENT_PROD_WRITE_KEY: '-' # A fake key just to make the build succeed
+
+      - name: Upload artifact prep-build-test-webpack
+        uses: actions/upload-artifact@v4
+        with:
+          name: prep-build-test-webpack
+          path: ./dist/
+
   run-tests:
     name: Run tests
     uses: ./.github/workflows/run-tests.yml
 
+  run-benchmarks:
+    uses: ./.github/workflows/run-benchmarks.yml
+    needs:
+      - prep-build-test-browserify
+      - prep-build-test-webpack
+
   wait-for-circleci-workflow-status:
     name: Wait for CircleCI workflow status
     uses: ./.github/workflows/wait-for-circleci-workflow-status.yml
@@ -112,6 +164,8 @@ jobs:
     secrets:
       PR_COMMENT_TOKEN: ${{ secrets.PR_COMMENT_TOKEN }}
 
+  # Explanation for all-jobs-completed and all-jobs-pass being separate:
+  # https://github.com/MetaMask/metamask-module-template/pull/151
   all-jobs-completed:
     name: All jobs completed
     runs-on: ubuntu-latest

diff --git a/.github/workflows/publish-prerelease.yml b/.github/workflows/publish-prerelease.yml
@@ -17,7 +17,7 @@ jobs:
           fetch-depth: 0 # This is needed to get merge base to calculate bundle size diff
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Get merge base commit hash
         id: get-merge-base

diff --git a/.github/workflows/run-benchmarks.yml b/.github/workflows/run-benchmarks.yml
@@ -0,0 +1,41 @@
+on:
+  workflow_call:
+
+env:
+  COMMANDS: |
+    {
+      pageload: 'yarn benchmark:chrome --out test-artifacts/chrome/benchmark/pageload.json --retries 2',
+      userActions: 'yarn user-actions-benchmark:chrome --out test-artifacts/chrome/benchmark/user_actions.json --retries 2',
+    }
+
+jobs:
+  benchmarks:
+    runs-on: ubuntu-22.04
+    strategy:
+      matrix:
+        buildType: [browserify, webpack]
+        testType: [pageload, userActions]
+    name: ${{ matrix.buildType }}-${{ matrix.testType }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Setup environment
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
+
+      - name: Download artifact prep-build-test-${{ matrix.buildType }}
+        uses: actions/download-artifact@v4
+        with:
+          path: ./dist/
+          pattern: prep-build-test-${{ matrix.buildType }}
+          merge-multiple: true
+
+      - name: Run the benchmark
+        # Choose a benchmark command from env.COMMANDS
+        run: ${{ fromJson(env.COMMANDS)[matrix.testType] }}
+
+      - name: Upload Artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: benchmark-${{ matrix.buildType }}-${{ matrix.testType }}
+          path: test-artifacts/chrome/benchmark/
diff --git a/.github/workflows/run-tests.yml b/.github/workflows/run-tests.yml
@@ -22,7 +22,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: test:unit:coverage
         run: yarn test:unit:coverage --shard=${{ matrix.shard }}/${{ strategy.job-total }}
@@ -43,7 +43,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: test:unit:webpack:coverage
         run: yarn test:unit:webpack:coverage
@@ -64,7 +64,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: test:integration:coverage
         run: yarn test:integration:coverage
@@ -92,12 +92,13 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Download artifacts
         uses: actions/download-artifact@v4
         with:
           path: coverage
+          pattern: coverage-*
           merge-multiple: true
 
       - name: Merge coverage reports

diff --git a/.github/workflows/test-circular-deps.yml b/.github/workflows/test-circular-deps.yml
diff --git a/.github/workflows/test-deps-audit.yml b/.github/workflows/test-deps-audit.yml
@@ -12,7 +12,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Run audit
         run: yarn audit
diff --git a/.github/workflows/test-deps-depcheck.yml b/.github/workflows/test-deps-depcheck.yml
@@ -12,7 +12,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Run depcheck
         run: yarn depcheck
diff --git a/.github/workflows/test-lint-changelog.yml b/.github/workflows/test-lint-changelog.yml
@@ -12,7 +12,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Validate changelog
         # For a `pull_request` event, the branch is `github.head_ref``.

diff --git a/.github/workflows/test-lint-lockfile.yml b/.github/workflows/test-lint-lockfile.yml
@@ -12,7 +12,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Lint lockfile
         run: yarn lint:lockfile

diff --git a/.github/workflows/test-lint.yml b/.github/workflows/test-lint.yml
@@ -12,7 +12,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Lint
         run: yarn lint

diff --git a/.github/workflows/test-storybook.yml b/.github/workflows/test-storybook.yml
@@ -12,7 +12,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Install Playwright browsers
         run: yarn exec playwright install chromium

diff --git a/.github/workflows/test-yarn-dedupe.yml b/.github/workflows/test-yarn-dedupe.yml
@@ -12,7 +12,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
 
       - name: Detect yarn lock deduplications
         run: yarn dedupe --check
diff --git a/.github/workflows/update-attributions.yml b/.github/workflows/update-attributions.yml
@@ -59,7 +59,7 @@ jobs:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           PR_NUMBER: ${{ github.event.issue.number }}
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
       - name: Get commit SHA
         id: commit-sha
         run: echo "COMMIT_SHA=$(git rev-parse --short HEAD)" >> "$GITHUB_OUTPUT"
@@ -81,7 +81,7 @@ jobs:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           PR_NUMBER: ${{ github.event.issue.number }}
       - name: Setup environment
-        uses: metamask/github-tools/.github/actions/setup-environment@main
+        uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
       - name: Generate Attributions
         run: yarn attributions:generate
       - name: Cache attributions file
@@ -182,4 +182,4 @@ jobs:
         env:
           GITHUB_TOKEN: ${{ secrets.LAVAMOAT_UPDATE_TOKEN }}
           PR_NUMBER: ${{ github.event.issue.number }}
-          ACTION_RUN_URL: "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+          ACTION_RUN_URL: '${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}'