perf: publish properties with the nightly dataset and measure cached Trino JSON (#1224)

* perf: publish properties with the nightly dataset and measure cached Trino JSON

One nightly frozen-perf run produced two unrelated datasets: the table
suite published as posthog-file-views-v1 and the properties suite as
properties-sha256-<inventory hash>, so the dashboard's dataset selector
showed one or the other and no single view compared the engines across
both.

Publish both suites under the nightly's dataset version and tell them
apart with a suite column (tables | properties) on runs and
query_results. The properties inventory hash moves to a fixture_version
column so history still never mixes fixtures. The publisher's schema
bootstrap adds both columns and classifies existing rows by the
-properties run ID suffix; a summary without a suite publishes as
tables, and an unknown suite is rejected.

Also measure the properties JSON queries on trino_cached ("trino
(cache)"). It only ran the unsupported VARIANT representation, so cached
Trino had no properties coverage; now that the Hoglake connector honors
fs.cache.enabled, the measurement is meaningful. Update the README
paragraph that still said the connector ignores the cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf: pair suites by nightly_run_id and migrate the schema ahead of data

Review fixes:

- Publish an explicit nightly_run_id (the table-suite run's ID) instead
  of leaving consumers to strip a -properties suffix from run IDs.
- Make the column migration one-shot: a guarded DO block adds suite,
  fixture_version and nightly_run_id and classifies older rows only when
  suite is missing, instead of two UPDATE scans on every publish.
- Add duckgres-perf-publisher --bootstrap-only and run it at the start
  of main scenario runs, so the schema (and a dashboard that depends on
  it) can land hours before the first data in the new shape; no window
  where the old dashboard picks the properties run as the latest run.
- Move the suite constants to core and validate the suite when the step
  starts, so a typo fails before the benchmark instead of at publish.
- Share the nightly dataset version through a YAML anchor.
- README: correct the uncached-cache cutoff and stop claiming that
  fixture_version alone keeps history from spanning fixture changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf: drop the bootstrap-only mode and the one-shot migration

Both solved problems that did not justify permanent machinery. The
bootstrap-only CLI mode and workflow step existed to close a one-time
window of a few hours where the dev perf dashboard's table panels would
be empty; merging the dashboard change right after the first publish
does the same. The guarded DO-block migration avoided two UPDATE scans
on a table that grows by ~200 rows a night; plain idempotent ALTERs and
a WHERE suite IS NULL backfill are simpler and just as correct.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Author: bill-ph

docs/runbooks/properties-perf.md

@@ -68,6 +68,7 @@ fixture catalogs and metadata without deleting source S3 files.
 | duckgres (vanilla) | Off | JSON |
 | duckgres (cache) | On | JSON |
 | trino (vanilla) | Off | JSON |
+| trino (cache) | On | JSON |
 | trino (cache+variant) | On | VARIANT — explicitly skipped |
 | Athena | — | STRUCT |
 
@@ -83,15 +84,15 @@ do not substitute JSON under a VARIANT label.
 
 ## Results and recovery
 
-Original results retain `perf/`, the original run ID, and dataset version.
-Properties use `perf-properties/`, a `-properties` run ID suffix, and an
-inventory-derived dataset version. Both summaries are handled by the existing
-publisher; only main-branch runs publish to the historical database. A
+Both suites publish under the nightly's dataset version with a `suite` column
+(`tables` / `properties`). Properties use `perf-properties/`, a `-properties`
+run ID suffix, and record the inventory-derived hash as `fixture_version`. Both
+summaries are handled by the existing publisher; only main-branch runs publish
+to the historical database. A
 properties failure still fails the scenario, but previously completed original
 results remain available for upload and publication. No timing result is
 claimed for a skipped comparison. Dashboard comparisons should filter
-`status = 'ok'` and use the matching properties intent; original aggregate
-panels retain their existing query selection.
+`status = 'ok'` and select the suite with the `suite` column.
 
 For missing/inaccessible prefixes, correct the selection or access and rerun.
 For Athena mapping failures, use the generated SQL and approved fixture

tests/mw-dev/scenario/perf/adapter_test.go

@@ -66,6 +66,10 @@ func TestExecutorRunsPerfStepAndWritesArtifacts(t *testing.T) {
 	if result.Summary.RunID != "scenario-run-1" || result.Summary.TotalQueries != 1 || result.Summary.TotalErrors != 0 {
 		t.Fatalf("summary = %+v", result.Summary)
 	}
+	// A perf step with no suite is the table suite and its own nightly.
+	if result.Summary.Suite != perfcore.SuiteTables || result.Summary.NightlyRunID != "scenario-run-1" {
+		t.Fatalf("summary suite/nightly = %q/%q", result.Summary.Suite, result.Summary.NightlyRunID)
+	}
 	pgwireDSN := factory.pgwireConnection.DSN
 	if !strings.Contains(pgwireDSN, "host=scenario-org.dev.example") || !strings.Contains(pgwireDSN, "password=root-password") {
 		t.Fatalf("pgwire dsn = %q, want scenario org host and provision password", pgwireDSN)
@@ -99,6 +103,36 @@ func TestExecutorRunsPerfStepAndWritesArtifacts(t *testing.T) {
 	}
 }
 
+func TestExecutorStampsSuiteMetadataAndRejectsUnknownSuites(t *testing.T) {
+	targets := []perfcore.Protocol{perfcore.ProtocolPGWire}
+	run := func(extra map[string]any) (*Executor, error) {
+		executor := NewExecutor(ExecutorConfig{
+			Connection: scenariosql.ConnectionConfig{DialHost: "127.0.0.1", SNISuffix: ".example.test", SSLMode: "require"},
+			OutputDir:  t.TempDir(), DriverFactory: &fakeDriverFactory{},
+		})
+		with := map[string]any{
+			"org_id": "scenario-org", "username": "root", "password": "test-password",
+			"catalog_file": writePerfCatalog(t, targets), "run_id": "nightly-1-properties",
+		}
+		for key, value := range extra {
+			with[key] = value
+		}
+		return executor, executor.ExecuteStep(context.Background(), core.Step{ID: "properties", Type: StepTypePerfQueries, With: with})
+	}
+	executor, err := run(map[string]any{"suite": "properties", "fixture_version": "properties-sha256-abc", "nightly_run_id": "nightly-1"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	result, _ := executor.State().Result("properties")
+	if result.Summary.Suite != perfcore.SuiteProperties || result.Summary.FixtureVersion != "properties-sha256-abc" || result.Summary.NightlyRunID != "nightly-1" {
+		t.Fatalf("summary = %+v", result.Summary)
+	}
+	// A typo fails before any measurement, not hours later at publish time.
+	if _, err := run(map[string]any{"suite": "property"}); err == nil || !strings.Contains(err.Error(), "unknown suite") {
+		t.Fatalf("expected unknown suite error, got %v", err)
+	}
+}
+
 func TestExecutorRecordsBothPGWireCacheVariantsWithEqualResources(t *testing.T) {
 	targets := []perfcore.Protocol{perfcore.ProtocolPGWireUncached, perfcore.ProtocolPGWireCached}
 	factory := &fakeDriverFactory{}

tests/mw-dev/scenario/perf/steps.go

@@ -71,6 +71,9 @@ type stepSpec struct {
 	Targets              []perfcore.Protocol
 	RunID                string
 	DatasetVersion       string
+	Suite                string
+	FixtureVersion       string
+	NightlyRunID         string
 	Database             string
 	OutputSubdir         string
 	ReadOnly             bool
@@ -203,6 +206,9 @@ func (e *Executor) ExecuteStep(ctx context.Context, step core.Step) error {
 		RunID:          spec.RunID,
 		Catalog:        catalog,
 		DatasetVersion: spec.DatasetVersion,
+		Suite:          spec.Suite,
+		FixtureVersion: spec.FixtureVersion,
+		NightlyRunID:   spec.NightlyRunID,
 		Drivers:        drivers,
 		Sink:           closingSink{sink: sink, closeFunc: closeSink},
 		Now:            e.now,
@@ -250,6 +256,10 @@ func (e *Executor) parseStep(step core.Step) (stepSpec, error) {
 	if err != nil {
 		return stepSpec{}, err
 	}
+	suite := stringFromWith(step, "suite", perfcore.SuiteTables)
+	if err := perfcore.ValidateSuite(suite); err != nil {
+		return stepSpec{}, classified(ErrorClassConfig, err)
+	}
 	if e.outputDir == "" {
 		return stepSpec{}, classified(ErrorClassConfig, fmt.Errorf("perf output dir is required"))
 	}
@@ -298,6 +308,9 @@ func (e *Executor) parseStep(step core.Step) (stepSpec, error) {
 		Targets:             targets,
 		RunID:               runID,
 		DatasetVersion:      stringFromWith(step, "dataset_version", ""),
+		Suite:               suite,
+		FixtureVersion:      stringFromWith(step, "fixture_version", ""),
+		NightlyRunID:        stringFromWith(step, "nightly_run_id", runID),
 		Database:            stringFromWith(step, "catalog", "ducklake"),
 		OutputSubdir:        stringFromWith(step, "output_subdir", "perf"),
 		ReadOnly:            boolFromWith(step, "read_only", true),

tests/mw-dev/scenario/runner_properties_test.go

@@ -14,6 +14,7 @@ import (
 	"github.com/posthog/duckgres/tests/mw-dev/scenario/core"
 	scenarioperf "github.com/posthog/duckgres/tests/mw-dev/scenario/perf"
 	scenariosql "github.com/posthog/duckgres/tests/mw-dev/scenario/sql"
+	perfcore "github.com/posthog/duckgres/tests/perf/core"
 	"github.com/posthog/duckgres/tests/perf/properties"
 	"gopkg.in/yaml.v3"
 )
@@ -23,6 +24,12 @@ const stepTypePropertiesComparison = "properties_comparison"
 // This phase runs only after the original benchmarks have written their
 // artifacts. Fixture preparation errors cannot prevent those results publishing.
 func (e dispatchExecutor) runPropertiesComparison(ctx context.Context, step core.Step) error {
+	// Properties results publish under the nightly's dataset so one dashboard
+	// selection covers every suite; the fixture hash is kept as fixture_version.
+	datasetVersion, _ := step.With["dataset_version"].(string)
+	if datasetVersion == "" {
+		return fmt.Errorf("properties comparison requires the nightly dataset_version")
+	}
 	source := os.Getenv("DUCKGRES_SCENARIO_PROPERTIES_S3_URI")
 	prepared, err := preparePropertiesComparison(ctx, source, step)
 	if err != nil {
@@ -43,13 +50,18 @@ func (e dispatchExecutor) runPropertiesComparison(ctx context.Context, step core
 	}}); err != nil {
 		return err
 	}
-	with := make(map[string]any, len(step.With)+3)
+	with := make(map[string]any, len(step.With)+6)
 	for key, value := range step.With {
 		with[key] = value
 	}
 	with["catalog_file"] = filepath.Join(prepared.directory, "catalog.yaml")
-	with["dataset_version"] = "properties-sha256-" + prepared.dataset.SHA256
+	with["dataset_version"] = datasetVersion
+	with["suite"] = perfcore.SuiteProperties
+	with["fixture_version"] = "properties-sha256-" + prepared.dataset.SHA256
 	with["output_subdir"] = "perf-properties"
+	// Distinct run ID so the publisher never overwrites the table-suite run;
+	// consumers pair the two through nightly_run_id, not this suffix.
+	with["nightly_run_id"] = fmt.Sprint(step.With["run_id"])
 	with["run_id"] = fmt.Sprint(step.With["run_id"]) + "-properties"
 	return e.perf.ExecuteStep(ctx, core.Step{ID: step.ID, Type: scenarioperf.StepTypePerfQueries, With: with})
 }

tests/mw-dev/scenario/runner_test.go

@@ -554,6 +554,11 @@ func TestFrozenPerfScenarioRunsOptionalPropertiesAfterOriginalPerf(t *testing.T)
 	if got := comparison.DependsOn; len(got) != 1 || got[0] != "perf_queries" {
 		t.Fatalf("properties dependencies = %#v, want original perf first", got)
 	}
+	// One nightly publishes both suites under one dataset, so the dashboard's
+	// comparison sees them together.
+	if got, want := comparison.With["dataset_version"], steps["perf_queries"].With["dataset_version"]; got == nil || got != want {
+		t.Fatalf("properties dataset_version = %#v, want the perf_queries dataset %#v", got, want)
+	}
 	for _, name := range scenario.RequiredEnv {
 		if name == "DUCKGRES_SCENARIO_PROPERTIES_S3_URI" {
 			t.Fatal("properties fixture must remain optional")

tests/mw-dev/scenario/scenarios/posthog_frozen_perf.yaml

@@ -111,7 +111,7 @@ steps:
       athena_poll_interval: 500ms
       athena_query_timeout: 30m
       run_id: ${run_id}
-      dataset_version: posthog-file-views-v1
+      dataset_version: &nightly_dataset posthog-file-views-v1
       fail_on_query_errors: true
       worker_cpu: ${env:DUCKGRES_K8S_WORKER_CPU_REQUEST}
       worker_memory: ${env:DUCKGRES_K8S_WORKER_MEMORY_REQUEST}
@@ -139,6 +139,8 @@ steps:
       athena_poll_interval: 500ms
       athena_query_timeout: 30m
       run_id: ${run_id}
+      # Same dataset as perf_queries: both suites of one nightly publish together.
+      dataset_version: *nightly_dataset
       output_subdir: perf-properties
       fail_on_query_errors: true
       worker_cpu: ${env:DUCKGRES_K8S_WORKER_CPU_REQUEST}

tests/perf/README.md

@@ -260,11 +260,11 @@ queries/iterations until eviction or teardown; one warm-up does not guarantee
 every replica holds the complete working set. Worker CPU/memory limits remain
 three workers at 1 CPU/4Gi per cluster; cache volumes reserve additional storage.
 
-The current frozen suite uses the pinned Hoglake connector, which ignores
-`fs.cache.enabled`; actual filesystem-cache support remains deferred. Thus
-`trino_cached` currently labels the requested configuration, not verified cache
-hits or a demonstrated cache benefit. Consolidating the scenario does not change
-that connector behavior. The baseline also does not guarantee cold JVM, OS, or
+The frozen suite runs the newest PostHog/trino master build, whose Hoglake
+connector honors `fs.cache.enabled` through Trino's shared filesystem module
+(PostHog/trino#43). Builds before that change ignored the property, so every
+`trino_cached` result from the pinned build, through the 2026-09-23 08:35 UTC
+nightly, measured an uncached cluster under the cached label. The baseline also does not guarantee cold JVM, OS, or
 storage-service caches. Keep the protocol labels separate in history.
 
 Trino `physicalInputBytes` can include bytes served from cache. Use cache-manager
@@ -287,22 +287,33 @@ iterations. Merely filtering a large mixed-day file set does not guarantee small
 scans. The runner does not generate data or enforce a row-count limit.
 
 The properties catalog measures JSON with `duckgres (vanilla)`, `duckgres (cache)`,
-and `trino (vanilla)`, plus STRUCT with Athena. Athena executes only STRUCT;
+`trino (vanilla)` and `trino (cache)`, plus STRUCT with Athena, so every engine
+covers every properties intent. Athena executes only STRUCT;
 its complete ordered results must match the shared Duckgres/Trino JSON baseline
 for each intent before properties measurements start. `trino (cache+variant)` is explicitly
 unsupported until Hoglake supports VARIANT: its two comparisons emit `skipped`
-rows with a reason and no timings, and do not connect to cached Trino. Skipped
-rows use iteration zero and are excluded from measured/warmup query counts.
-Original cached Trino benchmarks continue running normally.
-
-Original results retain `perf/`, the scenario run ID, and
-`posthog-file-views-v1`. Properties results use `perf-properties/`, a
-`-properties` run ID suffix, and a version derived from the selected object
-inventory. This keeps the original dashboard history stable and prevents the
-publisher from overwriting one result set with the other. Both use the existing
-publisher; only main-branch runs publish to the shared database. Dashboard
-comparisons must match properties intent IDs and successful statuses; existing
-full-corpus aggregate panels do not automatically include properties queries.
+rows with a reason and no timings. Skipped rows use iteration zero and are
+excluded from measured/warmup query counts.
+
+Both suites of one nightly publish under the same dataset version
+(`posthog-file-views-v1`, shared through a YAML anchor in the scenario), with
+three columns on `runs` and `query_results`:
+
+- `suite` (`tables` | `properties`, a closed set in `core`, validated when the
+  step starts so a typo fails before hours of measurement);
+- `nightly_run_id`, the table-suite run's ID, which pairs a nightly's suites
+  explicitly (a standalone run is its own nightly);
+- `fixture_version`, the hash of the selected properties object inventory.
+
+The properties suite keeps its own `perf-properties/` directory and a distinct
+`-properties` run ID, so the publisher never overwrites one result set with the
+other. `fixture_version` records which fixture each properties run measured; it
+does not by itself stop a history chart from spanning a fixture change, so
+regenerating the fixture is a deliberate history break. The publisher's schema
+bootstrap adds the columns and classifies rows published before they existed
+(by the `-properties` suffix they used to carry); rows with explicit values are
+never touched. A summary without a suite publishes as `tables`. Only
+main-branch runs publish to the shared database.
 
 All properties preparation happens after the original result files are complete,
 so a properties setup or validation failure cannot prevent their publication.

tests/perf/core/runner.go

@@ -24,6 +24,9 @@ type RunnerConfig struct {
 	RunID          string
 	Catalog        Catalog
 	DatasetVersion string
+	Suite          string
+	FixtureVersion string
+	NightlyRunID   string
 	Drivers        map[Protocol]ProtocolDriver
 	Sink           ResultSink
 	OnSetup        func(context.Context) error
@@ -57,6 +60,9 @@ func (r *QueryRunner) Run(ctx context.Context) (RunSummary, error) {
 	summary := RunSummary{
 		RunID:          runID,
 		DatasetVersion: r.cfg.DatasetVersion,
+		Suite:          r.cfg.Suite,
+		FixtureVersion: r.cfg.FixtureVersion,
+		NightlyRunID:   r.cfg.NightlyRunID,
 		StartedAt:      startedAt,
 		FinishedAt:     startedAt,
 	}

tests/perf/core/runner_test.go

@@ -82,6 +82,29 @@ func summaryProtocolIterations(sink *inMemorySink) []string {
 	return got
 }
 
+func TestRunnerStampsSuiteAndFixtureVersionOnTheSummary(t *testing.T) {
+	runner := NewQueryRunner(RunnerConfig{
+		RunID:          "run-properties",
+		DatasetVersion: "posthog-file-views-v1",
+		Suite:          "properties",
+		FixtureVersion: "properties-sha256-abc",
+		Catalog: Catalog{
+			Name: "suite", MeasureIterations: 1, Targets: []Protocol{ProtocolPGWire},
+			Queries: []Query{{QueryID: "q1", IntentID: "i1", PGWireSQL: "SELECT 1"}},
+		},
+		Drivers: map[Protocol]ProtocolDriver{ProtocolPGWire: &testDriver{protocol: ProtocolPGWire}},
+		Sink:    &inMemorySink{},
+		Now:     func() time.Time { return time.Unix(1700000000, 0) },
+	})
+	summary, err := runner.Run(context.Background())
+	if err != nil {
+		t.Fatalf("Run returned error: %v", err)
+	}
+	if summary.DatasetVersion != "posthog-file-views-v1" || summary.Suite != "properties" || summary.FixtureVersion != "properties-sha256-abc" {
+		t.Fatalf("summary = %+v", summary)
+	}
+}
+
 func TestRunnerExecutesPairedQueriesThroughExistingRuntimeContract(t *testing.T) {
 	catalog, err := ParseCatalog([]byte(pairedCatalogYAML(`
 paired_queries:

tests/perf/core/types.go

@@ -1,6 +1,7 @@
 package core
 
 import (
+	"fmt"
 	"strings"
 	"time"
 )
@@ -26,6 +27,8 @@ func (p Protocol) RunLabel(representation string) string {
 		return "duckgres (cache)"
 	case p == ProtocolTrino && representation == "json":
 		return "trino (vanilla)"
+	case p == ProtocolTrinoCached && representation == "json":
+		return "trino (cache)"
 	case p == ProtocolTrinoCached && representation == "variant":
 		return "trino (cache+variant)"
 	default:
@@ -113,14 +116,38 @@ type QueryResult struct {
 	ServiceMetrics   *ServiceMetrics `json:"service_metrics,omitempty"`
 }
 
+// Suites published under one dataset. Dashboards filter on this closed set.
+const (
+	SuiteTables     = "tables"
+	SuiteProperties = "properties"
+)
+
+// ValidateSuite rejects a suite outside the closed set, so a typo fails when a
+// run starts rather than after hours of measurement at publish time.
+func ValidateSuite(suite string) error {
+	if suite != SuiteTables && suite != SuiteProperties {
+		return fmt.Errorf("unknown suite %q (want %q or %q)", suite, SuiteTables, SuiteProperties)
+	}
+	return nil
+}
+
 type RunSummary struct {
-	RunID          string    `json:"run_id"`
-	DatasetVersion string    `json:"dataset_version"`
-	StartedAt      time.Time `json:"started_at"`
-	FinishedAt     time.Time `json:"finished_at"`
-	TotalQueries   int       `json:"total_queries"`
-	TotalErrors    int       `json:"total_errors"`
-	WarmupQueries  int       `json:"warmup_queries"`
+	RunID          string `json:"run_id"`
+	DatasetVersion string `json:"dataset_version"`
+	// Suite names the comparison family inside a dataset ("tables" or
+	// "properties"); one nightly publishes several suites under one dataset.
+	Suite string `json:"suite,omitempty"`
+	// FixtureVersion identifies the exact fixture a suite measured when it is
+	// finer-grained than the dataset (e.g. the properties object inventory).
+	FixtureVersion string `json:"fixture_version,omitempty"`
+	// NightlyRunID names the run every suite of one nightly belongs to: the
+	// table-suite run's ID. Consumers pair suites by it rather than by run IDs.
+	NightlyRunID  string    `json:"nightly_run_id,omitempty"`
+	StartedAt     time.Time `json:"started_at"`
+	FinishedAt    time.Time `json:"finished_at"`
+	TotalQueries  int       `json:"total_queries"`
+	TotalErrors   int       `json:"total_errors"`
+	WarmupQueries int       `json:"warmup_queries"`
 }
 
 // SQLFor preserves canonical SQL except for the JSON scalar extractor in the