Skip to content

scenario-dev.md

Latest commit

 

History

History
148 lines (122 loc) · 7.8 KB

scenario-dev.md

File metadata and controls

148 lines (122 loc) · 7.8 KB
Raw

Dev Scenario Runner

The scenario-dev GitHub Actions workflow runs one selected scenario against an isolated Duckgres stack in mw-dev. It uses the same tests/mw-dev harness as e2e-mw-dev for cluster access, isolated namespace deployment, pod identity, diagnostics, and teardown.

Scheduled Runs

The daily scheduled trigger runs posthog_frozen_perf, which creates production-shaped DuckLake tables from the frozen dataset, executes equivalent PGWire, Trino, and on-demand Athena queries, and publishes the comparison history. Scheduled runs no longer execute full-suite.

Manual runs remain selectable and default to full-suite, which covers the frozen dataset metadata, legacy perf, and dbt workloads with one shared warehouse lifecycle.

Scenario jobs override the shared harness's default worker request to 3 CPU and 12Gi memory for the frozen PGWire perf workload. The isolated Trino workers use the same aggregate execution resources. This adds process headroom for repeated full-dataset aggregates without changing DuckDB's thread count. The shared e2e workflow keeps the harness defaults documented in tests/mw-dev/README.md.

To omit dbt, manually run the workflow with scenario set to fast-suite. The input directly names a YAML file under tests/mw-dev/scenario/scenarios/, without the .yaml suffix. This also makes targeted scenarios such as provision_rejection, provision_smoke, posthog_frozen_metadata, posthog_frozen_perf, and posthog_frozen_dbt individually runnable. Invalid or missing scenario names fail before a payload Job is created.

Required Repository Configuration

The workflow follows the e2e dev harness access pattern: GitHub OIDC assumes a dev-scoped AWS role, Tailscale reaches the private cluster API, and payload jobs execute inside Kubernetes.

Configure these repository secrets:

  • AWS_ECR_PUBLISH_IAM_ROLE
  • MW_DEV_ACCOUNT_ID
  • MW_DEV_TRINO_POD_IDENTITY_ROLE

Configure these repository variables:

  • TS_WIF_CLIENT_ID_MW_DEV
  • TS_WIF_AUDIENCE_MW_DEV
  • MW_DEV_SCENARIO_PERF_SECRET_ID

Athena requires no additional GitHub settings. Terraform publishes its configuration to the SSM String parameter /duckgres/perf/athena in the dev account and grants the workflow's existing OIDC role ssm:GetParameter on that parameter only. Apply this infrastructure before enabling Athena runs. For frozen perf, scripts/scenario_athena_config.sh loads and validates the JSON after AWS authentication and before deploying the isolated stack. Other scenarios do not load it. The parameter contains pod_identity_role_arn, workgroup_name, glue_database_name, and results_s3_uri, derived from Terraform resources; they are configuration identifiers, not credentials.

The Athena values must identify an engine-v3 on-demand workgroup, a Glue database with explicit events and persons external tables over the frozen Parquet prefixes, and an encrypted S3 result root ending in /. Disable workgroup result reuse and configure lifecycle expiry on the result root. The scenario still disables result reuse on every query and validates that the effective result location remains under the configured root.

MW_DEV_SCENARIO_PERF_SECRET_ID names the AWS Secrets Manager JSON secret used only by historical perf publishing. The secret must contain host, port, database, username, and password. The workflow pipes it directly from AWS CLI into the publisher process so the password is not placed in a shell argument, environment variable, or log.

The workflow hardcodes the stable mw-dev cluster name/context and builds the control-plane pod identity role ARN from MW_DEV_ACCOUNT_ID, matching e2e-mw-dev. The scenario payload uses the isolated control-plane ClusterIP service and the per-run internal secret generated by tests/mw-dev/run.sh. The harness also injects DUCKGRES_SCENARIO_ORG_ID=ci-pr-<workflow-run-id>-cnpg for successful CNPG scenarios. That name matches the Crossplane composition's exact CI credential ownership boundary and the harness's cleanup convention.

The scenario Job uses the dedicated duckgres-scenario ServiceAccount. Its Pod Identity role needs only Athena query APIs for the configured workgroup, read-only Glue metadata access to the benchmark database, read/list access to the frozen dataset, and result-prefix access (plus the configured KMS key when SSE-KMS is used). The workflow deployer needs iam:PassRole/iam:GetRole for that role. Teardown and the stale-run sweep already delete every Pod Identity association in the disposable namespace.

For a direct local run with just scenario, set DUCKGRES_SCENARIO_ORG_ID explicitly to an authorized, disposable org id. The local runner has no default because reusing a fixed warehouse identity can collide with another run. Scenario-specific frozen-dataset variables remain required as reported by scripts/scenario_run.sh --check-env.

Failure Recovery

If loading /duckgres/perf/athena fails, confirm the Athena Terraform apply completed, the workflow is using the expected AWS account/region, and its OIDC role has the exact-parameter read grant. Invalid or incomplete JSON fails before deployment and exports no partial settings. Fix the Terraform source and apply it again rather than editing the parameter manually. Rerun the workflow after the parameter and permission are available.

Every scenario sub-run provisions its own org or warehouse name and includes cleanup steps. The isolated stack also tears down the temporary namespace after the payload finishes. The harness keeps the artifact volume attached to a lightweight sidecar until the runner copies it; a missing or failed copy fails the scenario job instead of being silently ignored. Incomplete copies remain uploadable in a visible *.partial/ directory with an artifact_collection_error.txt marker. When a run fails, inspect the uploaded scenario-dev-* artifact first:

  • scenario_summary.json contains the overall run verdict.
  • scenario_summary.md highlights failed and skipped steps and is also published in the GitHub Actions job summary.
  • step_results.csv contains per-step duration, status, and error class.
  • events.jsonl contains the execution timeline.
  • perf/query_service_metrics.csv separates Athena queue/planning/engine time and records bytes scanned for cost analysis.

Perf query errors mark the perf_queries DAG step failed. They do not stop independent sibling branches: for example, dbt_models still runs because it depends on setup_frozen_views, not perf_queries. Only true dependants are skipped, and always_run teardown still executes. The final workflow result is reported after all eligible steps and artifact collection finish.

After teardown, scheduled and manually dispatched runs on main publish any collected perf/summary.json and perf/query_results.csv into the persistent scenario perf result schema. Publishing runs with if: always() so measured query failures remain visible. Runs without a perf artifact skip publishing, and non-main branch runs never publish into the shared history. Each artifact has a two-minute publish timeout so one stalled database operation cannot prevent later artifacts from being attempted; the workflow step retains a ten-minute overall timeout.

If cleanup did not complete, the scenario-created org is ci-pr-<workflow-run-id>-cnpg; deprovision it manually through the relevant dev control plane. Also inspect the workflow diagnostics from tests/mw-dev/run.sh diagnostics; the namespace is duckgres-ci-pr-<workflow-run-id>.

If an Athena query times out, the driver calls StopQueryExecution. A hard pod termination can still leave result objects, so S3 lifecycle expiry is the reliable cleanup backstop. Check the Athena query history for scan-limit or catalog errors, confirm the workgroup did not override results outside the configured root, and rerun after fixing the private Glue/IAM configuration.