[DO NOT MERGE] Integration preview: range replay + kernel replay perf stack on develop

projects/rocprofiler-sdk/CHANGELOG.md

@@ -24,6 +24,11 @@ Full documentation for ROCprofiler-SDK is available at [rocm.docs.amd.com/projec
 
 - rocPD schema 3.0.5 stores SPM counter location on `rocpd_info_pmc`. Each agent and combination of counter, XCC, shader engine, shader array, WGP, and instance has its own pmc id. Location dimensions not provided for a counter are stored as `NULL` and left blank in CSV output. The instance index is always stored. `rocpd_pmc_event` keeps the sample value and points at that id.
 - Request ROCr high-precision timestamp correlation during HSA registration when GPU timestamp collection is configured and the runtime provides the tools API. Older runtimes and tools that only trace CPU APIs or markers retain the existing behavior.
+**API:**
+
+  - Kernel replay keeps its snapshot staging memory between replayed dispatches instead of allocating, zero-filling, and freeing the whole tracked footprint on every dispatch. What is kept between dispatches is bounded by the most recent snapshot's footprint.
+  - Kernel replay takes its drain signal from the SDK's signal pool and no longer snapshots HIP's one-byte `__hip_cuid_*` compilation-unit markers.
+  - Each replayed dispatch logs its lock and drain, snapshot, pass, and restore times at `ROCPROFILER_LOG_LEVEL=info`.
 
 ### Resolved issues
 
@@ -84,6 +89,7 @@ Full documentation for ROCprofiler-SDK is available at [rocm.docs.amd.com/projec
 
 ### Resolved issues
 
+  - `rocprofv3 --kernel-replay-beta-enabled` given without `--replay-mode kernel` is now rejected instead of ignored. The flag acknowledges that kernel replay is a beta feature; it does not select it. Its name reads as if it did, so a command line carrying only the acknowledgement silently ran application replay, and the counter values that produces are indistinguishable from replayed ones.
   - Fixed `rocprofv3` crashing during output generation when a second tool subscribed to code object tracing in the same process, which blocked profiling PyTorch and Triton workloads through rocprofiler-compute.
   - Fixed `rocprofv3` hanging instead of exiting when a fatal signal arrives while it is already handling one, for example when output generation aborts. It previously left GPU child processes running and required killing the process manually.
   - Fixed memory write-bandwidth telemetry reporting N/A on gfx1250 (MI455X) by adding the missing `WRITE_SIZE` derived counter, computed from the `GL2C_WRITE_SECTORS` hardware counter. The read-side `FETCH_SIZE` counter was already defined for gfx1250 while the write side was not, so downstream consumers such as the memory write-bandwidth field exposed through RDC reported N/A.

projects/rocprofiler-sdk/benchmark/CMakeLists.txt

@@ -103,3 +103,4 @@ endif()
 include(timem)
 
 add_subdirectory(source)
+add_subdirectory(tests)

projects/rocprofiler-sdk/benchmark/README.md

@@ -29,14 +29,34 @@ times, snapshotting and restoring device memory around each pass. Application re
 application N times instead. Which is cheaper depends entirely on the workload, so this
 configuration is built to find the crossover rather than to assert a number.
 
+Passing several `--pmc` flags without kernel replay selects application replay: rocprofv3 runs the
+application once per group. Counter rotation is a separate mode configured through an input file's
+`pmc_groups` and `pmc_group_interval` fields; it runs the application once but no dispatch carries
+every group, so it is not a like-for-like comparison.
+
 The jobs vary the three terms of the cost model — `dispatches x tracked footprint x passes` —
 independently, so a result can be attributed to one of them instead of to the workload as a whole.
-The headline comparison is between two named configurations collecting the same four counters:
+A fourth sweep varies how many dispatches the application runs between synchronizations, which is
+the axis range replay will be measured on once it exists.
+
+A single rocprofv3 invocation can measure application replay directly: multiple `--pmc` groups
+cause one full application run per group. The single-group configuration is retained as a reference,
+and the `benchmark_replay_*` views multiply its additive metrics by the group count to project the
+total. For the four-group measured and projected comparison:
 
 ```bash
-rocprofv3-benchmark -i ./kernel-replay.yaml --filter-rocprofv3 replay_4_groups app_replay
+rocprofv3-benchmark -i ./kernel-replay.yaml --filter-rocprofv3 replay_4_groups application_replay_4_groups counters_only
+```
+
+```sql
+SELECT benchmark_label, counter_collection_mode, measured, application_replay_projected
+FROM benchmark_replay_wall_time;
 ```
 
+`scripts/replay_perf.py` measures application replay directly, input-file counter rotation, and
+kernel replay, and also covers the cost replay imposes on dispatches that opt out. It drives rocprofv3
+itself, so it needs no database and is run by hand rather than in CI.
+
 One job is expected to show almost no overhead: `replay-hip-graph-declines`. A HIP graph launch is
 not a single-packet single-dispatch submission, so replay declines it. That job exists to make the
 absence visible here rather than in production, since a graph-capturing application gets no replay

projects/rocprofiler-sdk/benchmark/kernel-replay.yaml

@@ -1,10 +1,25 @@
 # Kernel replay overhead benchmark configuration.
 #
-# Kernel replay collects N counter groups from a single application run by re-executing each
-# dispatch N times, snapshotting and restoring the agent's tracked device memory around each pass.
-# The alternative it replaces -- application replay -- runs the whole application N times, once per
-# group. Which one is cheaper is entirely a property of the workload, and this configuration is
-# built to measure where the crossover is rather than to assert a number.
+# There are three ways to collect N counter groups, and they do not collect the same thing:
+#
+#   application replay  run the whole application N times, once per group. Every dispatch ends up
+#                       with every group, at the cost of N application runs. This is the thing
+#                       kernel replay is meant to replace.
+#   multiplexed         run the application once and rotate groups across dispatches using the
+#                       input-file pmc_groups / pmc_group_interval options. No dispatch carries
+#                       every group.
+#   kernel replay       run the application once and re-execute each dispatch N times,
+#                       snapshotting and restoring the agent's tracked device memory around each
+#                       pass. Every dispatch ends up with every group, from one run.
+#
+# Only application replay and kernel replay produce the same data, so only those two are a
+# like-for-like comparison. Which of the two is cheaper is entirely a property of the workload, and
+# this configuration is built to measure where the crossover is rather than to assert a number.
+#
+# One benchmark job is one rocprofv3 invocation, and multiple CLI --pmc groups make that invocation
+# run the application once per group. application_replay_4_groups below measures that cost directly.
+# The single-group job is also retained as a reference: benchmark_replay_* projects its cost across
+# the group count when that estimate is useful.
 #
 # The cost model the jobs are laid out against is:
 #
@@ -21,15 +36,15 @@
 #
 # Run with:
 #   rocprofv3-benchmark.py -i kernel-replay.yaml
-#   rocprofv3-benchmark.py -i kernel-replay.yaml --filter-rocprofv3 replay_4_groups app_replay
+#   rocprofv3-benchmark.py -i kernel-replay.yaml --filter-rocprofv3 replay_4_groups application_replay_4_groups counters_only
 #
 # See docs/conceptual/kernel_replay/kernel_replay_performance.md for what the numbers mean.
 
 defaults:
   rocprofv3:
-    # The comparison that matters. Both collect the same four counter groups; replay_4_groups does
-    # it in one application run, app_replay in four. Filter to just these two for the headline
-    # number.
+    # The comparison that matters. replay_4_groups collects four groups on every dispatch in one
+    # application run; application_replay_4_groups runs the application once per group. The
+    # counters_only reference also lets the view project application replay from a single-group run.
     replay_4_groups:
       [
         --replay-mode,
@@ -44,7 +59,9 @@ defaults:
         --pmc,
         SQ_BUSY_CYCLES,
       ]
-    app_replay:
+    # The same four groups without kernel replay: rocprofv3's default application-replay mode runs
+    # the application once per group.
+    application_replay_4_groups:
       [--pmc, SQ_WAVES, --pmc, GRBM_COUNT, --pmc, SQ_INSTS_VALU, --pmc, SQ_BUSY_CYCLES]
 
     # Pass-count sweep at fixed footprint and dispatch count. Wall time should grow roughly
@@ -84,8 +101,12 @@ defaults:
         SQ_INSTS_SMEM,
       ]
 
-    # Counter collection without replay, as the floor: whatever this costs is attributable to
-    # counter collection rather than to the replay machinery.
+    # One counter group, no replay. Two jobs in one: the floor, so that whatever the replay
+    # configurations cost above it is attributable to the replay machinery rather than to counter
+    # collection, and the unit of application replay, since collecting G groups that way is this
+    # run repeated G times. The benchmark_replay_* views look for exactly this configuration when
+    # projecting what an application replay would have cost, so it needs to stay in any filtered
+    # run that is meant to produce a comparison.
     counters_only: [--pmc, SQ_WAVES]
 
   benchmark:
@@ -165,6 +186,40 @@ jobs:
     group: [kernel-replay, dispatch-sweep]
     command: [transpose, 1, 2000, 100, 2048, 2048]
 
+  # ---------------------------------------------------------------------------
+  # Dispatches-per-range axis: same dispatch count, footprint and pass count, but the application
+  # synchronizes after every nth dispatch instead of at a fixed cadence.
+  #
+  # transpose's third argument is how many dispatches it enqueues between stream synchronizations,
+  # which is the length of an uninterrupted run of dispatches. Kernel replay drains the agent per
+  # replayed dispatch, so an application that already synchronizes constantly has less to lose from
+  # that drain than one that keeps a deep queue in flight, and this sweep is what separates the two.
+  #
+  # It is also the axis range replay will be measured on. Range replay snapshots once per range
+  # rather than once per dispatch, which turns the cost model from
+  #
+  #   dispatches x footprint x passes    into    ranges x footprint x passes
+  #
+  # so the saving is proportional to how many dispatches fall inside a range. Recording the axis now
+  # means the kernel replay numbers it will be compared against already exist, measured on the same
+  # workloads, rather than having to be regenerated later against a moved baseline.
+  # ---------------------------------------------------------------------------
+  - name: replay-sync-every-dispatch
+    group: [kernel-replay, range-length-sweep]
+    command: [transpose, 1, 500, 1, 2048, 2048]
+
+  - name: replay-sync-every-10-dispatches
+    group: [kernel-replay, range-length-sweep]
+    command: [transpose, 1, 500, 10, 2048, 2048]
+
+  - name: replay-sync-every-100-dispatches
+    group: [kernel-replay, range-length-sweep]
+    command: [transpose, 1, 500, 100, 2048, 2048]
+
+  - name: replay-sync-once
+    group: [kernel-replay, range-length-sweep]
+    command: [transpose, 1, 500, 500, 2048, 2048]
+
   # ---------------------------------------------------------------------------
   # Application shapes.
   # ---------------------------------------------------------------------------