Skip to content

v1.91.102.0 fix(pregate): run Dart and Flutter test lanes - #3193

Closed
arlophoenix wants to merge 21 commits into
garrytan:mainfrom
arlophoenix:fix/pregate-dart
Closed

arlophoenix wants to merge 21 commits into
garrytan:mainfrom
arlophoenix:fix/pregate-dart

Conversation

@arlophoenix

@arlophoenix arlophoenix commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Why (in your own words)

gstack-pregate now works on Dart and Flutter repos. Before this, every touched file in a Flutter repo ended PREGATE_NO_LANE, so gstack-ship-receipt write --pregate refused on every ship. .dart was not a source extension, _test.dart was not a test and package: imports never resolved, and a lane declared in .gstack/pregate.json didn't help, because a declared edge only counts when it points at a recognised test. This branch resolves imports through every tracked pubspec.yaml, runs each package's tests from its own root with flutter test or dart test (through fvm when pinned), and treats Flutter and Dart CI jobs as lanes. A repo that keeps .gstack/ out of git can keep its policy in <GSTACK_HOME>/projects/<slug>/pregate.json, including local lanes for CI jobs a Mac can't otherwise clear. Separately, gstack-diff-scope matched no category for .dart, so a Flutter widget PR skipped the design, performance and security reviewers. .dart now counts as frontend and backend, and /review and /ship detect a Dart stack and test framework.

Live evidence

A throwaway local clone of a real Flutter app (flutter-bbtm), one commit touching test/ui/widgets/bbtm_drawer_widget_test.dart, and the user policy { "local": ["test.yml/flutter", "test.yml/guards", "qa-driver.yml/smoke"] } in a scratch GSTACK_HOME:

# before: installed gstack 1.91.101.0
$ gstack-pregate --stage tests --base develop --allow-repo-commands
tests      lanes     fail                157 ms  1 touched file(s) with no test lane: test/ui/widgets/bbtm_drawer_widget_test.dart (PREGATE_NO_LANE)

# after: this branch at f5f3cee5c
$ bin/gstack-pregate --stage tests --base develop --allow-repo-commands --explain
pregate: policy user …/projects/Development-flutter-bbtm/pregate.json; registry origin/develop@be22dc580bec (no .gstack/generated.json); base origin/develop@be22dc580bec; 1 touched file(s); platform darwin
tests      lanes     pass              14094 ms  14 tests ran in 1 file(s), exit 0
    lanes: Tiers ran: unit; requires-remote: none
    lanes: ran: fvm flutter test 'test/ui/widgets/bbtm_drawer_widget_test.dart'

Same clone, one commit touching lib/theme.dart, lib/widgets/bbtm_drawer/view/bbtm_drawer.dart and the drawer test:

$ gstack-diff-scope develop        # before, 1.91.101.0
SCOPE_FRONTEND=false SCOPE_BACKEND=false SCOPE_TESTS=true
$ bin/gstack-diff-scope develop    # after
SCOPE_FRONTEND=true SCOPE_BACKEND=true SCOPE_TESTS=true

On an earlier build of this branch, the import graph for the full repo recognised 984 tests in about 490 ms. lib/theme.dart selected 783 tests, and packages/cache/lib/cache.dart selected 743 through package:cache.

Scope

  • Changed: the pregate import graph, run groups, CI lane recognition (flutter test, dart test, dart run test, melos test, very_good test, with test paths resolved against each step's working-directory) and the user-owned policy with local. Device tests under integration_test/ and test_driver/ are named with a fix but never run, and a missing SDK is incomplete rather than exit 127. Also .dart in gstack-diff-scope and lib/frontend-scope.ts (gstack-design-detect skips .dart, since impeccable reads web sources), Dart stack and test-framework detection in /review and /ship, docs/pregate.md and docs/troubleshooting.md. To keep ship, review and plan-eng-review inside the parity-suite byte caps, the diff-size count became one git diff --numstat | awk line (same total), and some existing resolver prose was shortened.
  • Verified live by: the runs above, and test/pregate-dart.test.ts (15 tests covering the Dart graph, per-package run groups, an all-skipped run as ZERO-RUN, fvm pins, working directories, device tests and the user policy; PATH shims stand in for the SDK).
  • Did NOT test: Linux or Windows hosts. A wide selection end to end: lib/theme.dart selects 783 files, which may exceed the 600 s default and end incomplete. The pregate tests stage on this PR's own diff, which selects 596 files and ran past the 600 s cap on this Mac. Paid evals: the PR profile selects 108 probes and 24 judges, and a fork can't dispatch them.

bun run test on the final tree failed on this Mac: 367 failing tests across 91 files, mostly CSO, browse cookies, QA evidence and doctor, with other work loading the machine. I re-ran 30 of those files one at a time on this branch and on upstream/main. Every failure that reproduces alone fails with the same test names on upstream/main, and the rest pass alone. The other 61 files (1 to 6 failures each, none in code this branch touches) were not re-checked. Seven pregate tests in pregate-selection, pregate-invalidation and pregate-guards fail the same way on upstream/main because they assume a Linux host, where ubuntu-* lanes are tier unit.

Codex reviewed the whole branch once, and its two P1 findings are fixed. Fixes made after that pass were reviewed by Claude on the fix diff only.

Liveness proof (required for external contributors)

Screenshot attached by @arlophoenix:
image

Checklist

  • Liveness screenshot attached (GSTACK PR typed live into a real surface) or PR author is @garrytan (owner exemption)
  • This is not a generated-file-only diff (I edited the source/template and regenerated)
  • No ETHOS.md edits, and no changes to voice / founder perspective / YC references
  • New public command / external service / host adapter has an accepted issue linked (or N/A): N/A, no new command, service or adapter
  • Linked issue or reproduction: reproduction in Live evidence above (no tracked issue)

Not addressed

  • Editing the user policy after a run doesn't invalidate the recorded pregate run, because verify hashes repo inputs only.
  • On timeout only the bash wrapper is killed, so flutter test children can outlive it. An fvm pin whose SDK isn't installed falls back to the PATH SDK without saying so.
  • The receipt's runtime label is Bun, not the Dart SDK version. Duplicate Dart package names resolve to the first root, and zero-indented workflow step lists aren't parsed.
  • pubspec.lock in the evidence lockfiles, a Dart case in gstack-first-task-detect, pub support in /cso, a Dart option in the test-bootstrap runtime question, and per-glob runners are left for follow-ups.

Test plan

  • On the merged tree: test/pregate-dart.test.ts 15/15, plus diff-scope, frontend-scope parity, ship-test-detection-markers, gstack-design-detect, gen-skill-docs, skill-validation, touchfiles and the parity and budget suites, all passing
  • bun run typecheck clean; gstack-regen check fresh
  • bun run test full free suite: red on this Mac; see Scope for what was checked against upstream/main
  • Paid evals (eval:bg:pr): deferred, not dispatchable from a fork

🤖 Generated with Claude Code

arlophoenix and others added 21 commits October 14, 2026 09:25
A Dart repo got PREGATE_NO_LANE for every touched file: .dart was not a
source extension, _test.dart was not a test, and package: URIs never
resolved. Dart directives now resolve through every tracked pubspec.yaml,
and Dart tests run per package root with flutter/dart test (through fvm
when pinned), judged on the +N -M verdict line.

Two policy gaps blocked the same repos after mapping worked: a linux-only
CI lane is requires-remote from a Mac with no way to clear it when CI is
off (new `local` key), and a repo that gitignores .gstack/ could never
carry a base policy (fallback to <GSTACK_HOME>/projects/<slug>/pregate.json,
read only when the base has none; outside the repo, so a PR cannot reach it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dart lists

gstack-diff-scope matched no category for .dart, so a Flutter widget PR
with a test set SCOPE_TESTS alone: the design, performance and security
passes were skipped with no SCOPE_ERROR, because the test file matched.
The extension cannot separate a widget from a repository, so .dart is both
(lib/frontend-scope.ts mirrors the frontend half; impeccable still scans
web sources only). pubspec.lock is config.

pregate's workflow and selection-manifest parsers now read _test.dart
paths, and pubspec.yaml/.lock, analysis_options.yaml and dart_test.yaml
are conventional root files, not strays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etection

The review-army stack hint and TEST_FW knew Ruby, Node, Python, Go and
Rust, so a Flutter repo was described to every specialist as "node" (from
its package.json) with an unknown test framework. Test bootstrap had no
Dart runtime marker and its test-file count missed *_test.dart outside a
test/ directory. pubspec.yaml now reports dart or flutter (by the Flutter
SDK dependency), flutter-test/dart-test, RUNTIME:dart, FRAMEWORK:flutter
and dart_test.yaml, with flutter test / dart test (through fvm when
pinned) as the command to offer. Generated docs regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	bin/gstack-pregate
#	docs/pregate.md
#	lib/pregate/checks/lanes.ts
#	lib/pregate/config.ts
…policy paths

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…SDK, user-policy scope

CI jobs that run flutter test or dart test are test lanes. integration_test/
and test_driver/ are not auto-selected (they need a device). A missing
Dart SDK is incomplete and named, not a 127 test failure. The user-owned
pregate.json covers run only and refuses sealed/price_file, with a fix line
that points at the user file. planRuns groups linearly and memoises package
roots; the incomplete detail names the unclaimed files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, fvm hint only when pinned

A touched integration_test/ or test_driver/ file now gets a fix naming the
active policy file (the user-owned copy when that is the one in force) and the
`[]` declaration that clears it; the docs said to declare a lane, which could
never select a non-test. `dart run test`, `melos test` and `very_good test`
count as CI test commands. The missing-SDK hint offers fvm only when the repo
pins a version, and a null `price_file` in the user policy is not refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ir pattern

The fix text and docs now recommend "**/integration_test/**": [] (and
test_driver), which also matches a sub-package's device tests; the root-only
glob left them failing. Device detection lives in import-graph (any .dart file
under integration_test/ or test_driver/), shared by isTestFile and the remedy,
so driver entry points get the hint too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uses, per-run zero-run

`dart test` runs files under test/ only, so a production lib/src/ab_test.dart
is no longer selected and run as a test (a false lane failure). A user-owned
policy that exists but cannot be read now refuses instead of running on
defaults, which silently dropped its guards. One package whose tests all skip
makes the lane ZERO-RUN even when another package's count would hide it, and
{files} is substituted with a function so a `$&` in a file name stays literal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y error names the file

The test/ anchor let the manifest regex start inside a directory name, so
integration_test/app_test.dart became a phantom test/app_test.dart in that
lane's set. A left boundary keeps the match on the whole path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…est paths; guarded pubspec reads

- `+0 ~N: All tests skipped.` parses as 0 ran, so a fully skipped package is
  ZERO-RUN instead of a pass.
- Test paths a workflow step names resolve against the step's (or job's, or
  workflow's) working-directory, so a sub-package's platform lane keeps its
  obligation; a `- run: |` block no longer swallows a sibling key.
- Several Dart directives on one line are all recorded.
- pubspec.yaml reads skip non-regular files, escaping symlinks and files over
  1 MiB.
- fvm pins are found from the package up to the repo root, and flow-style
  `flutter: {sdk: flutter}` selects `flutter test`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y budgets

The Dart/Flutter stack and test-framework lines pushed ship, review and
plan-eng-review past their parity caps. Tighter prose and a one-line
`--numstat | awk` diff-size count replace the three-line `--stat | grep`
form (same total), so all three fit without raising a threshold. Regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lute/.. dirs unresolved; flutter sdk match skips comments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ubspec.yaml, as test infrastructure

Every Dart package has a pubspec.yaml, so listing it reported test
infrastructure for untested projects. Regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-directory and missing SDK in pregate and troubleshooting docs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CHANGELOG conflict: both sides added 1.91.101.0; this branch's entry moves to 1.91.102.0 above upstream's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@trunk-io

trunk-io Bot commented Oct 15, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@arlophoenix

Copy link
Copy Markdown
Contributor Author

/trunk merge

@trunk-io

trunk-io Bot commented Oct 15, 2026

Copy link
Copy Markdown

An error occurred while submitting your PR to the queue: Only users that are a part of this repo's Trunk organization or have write permissions to the repo can submit a PR to the queue

@capy-ai

capy-ai Bot commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

Thanks @arlophoenix, this is a substantial piece of work, and we'd like Dart and Flutter as a first-class pregate stack, including the user-owned policy file outside the repo.

Decision: yes, split into three PRs, each rebased on current main:

  1. Dart import graph and lanes.
  2. The user-owned pregate policy and local lanes.
  3. Diff-scope and review/ship stack detection.

Main's lanes now run through supervise() (#3209, #3190), so the runner changes need to move onto that. #3177 (bats) landed in #3236 and touches the same TEST_FILE_RE region, so rebase on top of it. Leave VERSION and CHANGELOG unstamped (we stamp at merge), and run bun test test/pregate-selection.test.ts plus your Dart tests. We'll land each piece with your authorship kept.

@arlophoenix

Copy link
Copy Markdown
Contributor Author

Split into three PRs as requested, each rebased on current main with VERSION and CHANGELOG unstamped:

  1. fix(pregate): Dart import graph and test lanes #3285 Dart import graph and test lanes
  2. feat(pregate): user-owned policy file and local lanes #3286 user-owned policy file and local lanes
  3. fix(scope,review,ship): recognise Dart and Flutter in diff scope and test detection #3284 Dart in diff-scope and review/ship test detection

Closing this one in their favour.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant