Skip to content

fix(replay): move replay vision and scoring schedules off minute zero - #106990

Draft
aspicer wants to merge 1 commit into
aspicer/jitter-basefrom
aspicer/jitter-replay
Draft

aspicer wants to merge 1 commit into
aspicer/jitter-basefrom
aspicer/jitter-replay

Conversation

@aspicer

@aspicer aspicer commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Replay vision's hourly read meter and search suggestion refresher start at minute zero, and its sub-hourly singletons fire on their interval boundaries with the rest of the fleet. Split from #106458; merge after the helper in #106984.

Changes

The hourly read meter and search suggestions now start two minutes after the hour with up to ten minutes of jitter. The estimates refresh, reconciler and Gemini cleanup sweep take a stable offset keyed by schedule id, and so does the surfacing scoring sweep. upsert_interval_schedule now requires an offset, so a new singleton cannot forget one.

Per-scanner offsets now come from the shared helper. They stay evenly spread, but map to different minutes. Existing scanner schedules keep their current offset until the scanner is next updated.

Replay count metrics keep minute zero on purpose. Their query reads a rolling hour with no cursor, so a later start would skip data. The schedule now says so, with an exemption for the minute-zero lint in #106997.

How did you test this code?

The replay vision schedule, read meter, reconciler, search suggestion and estimate tests pass locally, except the cases that need Postgres, which was down locally. The surfacing sweep schedule test passes. CI runs the full suites, and they passed with the same code on #106458.

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Release status

  • No feature flag controls this change
  • This change is behind a feature flag and is not available to users
  • This change makes a previously flagged feature available to everyone

Automatic notifications

  • Publish to changelog?

Docs update

None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5; Codex, GPT-6 (review fixes on #106458).

Split from #106458 by owning team: the owners.yaml resolver decided ownership, and a team gets its own PR when its owned files clear the reviewer assigner's bar of 10 lines or 3 files. The code is unchanged from the reviewed head of #106458, except that its signals test-fixture fix dropped because #106460 fixed the same bug on master. CodeRabbit CLI ran once with --deep over all the split PRs combined on master 8475a79 and reported no findings. Skills for the split: stacking-prs, establishing-code-ownership, reviewing-with-coderabbit, writing-pr-descriptions. The original change also used qa-team, announcing-behavior-changes, writing-tests, writing-code-comments, writing-ui-components, writing-user-facing-copy, running-ci-preflight and debugging-ci-failures.

Replay vision singleton schedules now pass an offset. The hourly read
meter and search suggestions start two minutes after the hour with ten
minutes of jitter, and the sub-hourly ones take a stable offset from the
shared helper, which compute_schedule_offset now uses too. The surfacing
scoring sweep takes a stable offset as well.

Replay count metrics keep minute zero because the query reads a rolling
hour with no cursor. The schedule carries a nosemgrep reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aspicer aspicer self-assigned this Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

✅ Duplication (Python) — clean

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

✅ Duplication (TypeScript) — clean

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

@greptile-apps

greptile-apps Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Retrigger

[Medium risk] Adjusts timing of scheduled replay and vision workflows.

The PR appears safe to merge, with a non-blocking gap in tests for its schedule timing.

Reviews (1) · Last reviewed commit: "fix(replay): move replay vision and scor..."

Comment thread products/replay_vision/backend/temporal/schedule.py
@coderabbitai

coderabbitai Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

📝 Walkthrough

Walkthrough

The replay-count metrics schedule keeps its hourly interval and now documents why it runs at minute zero. Replay Vision schedule offsets now use deterministic values, and the interval schedule helper requires an offset and supports optional jitter. The read-meter and search-suggestions schedules configure a two-minute offset and ten-minute jitter.

Priority: ⬇️ Low

Merge Risk: 🔵 Low · up to 409d6

Newly created scanner schedules may start together despite their staggered timing. This is limited to initial runs and has a localized correction; the remaining merge risk is low.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 409d6

The affected jobs retain their identities and existing update behavior, and the reviewed callers supply the required offsets. Production behavior when jitter is applied to existing schedules remains unverified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The timing changes reach recurring Replay Vision jobs and a session replay scoring sweep. The read meter aggregates across scanner rows, so a scheduling failure could affect the freshness of fleet-wide throttle inputs; no such failure was established.

Trust Boundaries and Controls

  • observed — The changed helper passes caller-supplied timing values to the Temporal schedule specification while retaining the existing workflow ID, task queue, overlap policy, and create/update paths. The inspected change does not establish a new attacker-controlled source or credential boundary.

Resilience and Maintainability Implications

  • observed — The read-meter activity rescans hour-aligned buckets and overwrites refreshed bucket values, providing a recovery path for delayed runs within its rescan window. Its runtime freshness under the new jitter was not measured.
🚥 Pre-merge checks | ✅ 1
✅ Passed checks (1 passed)
Check name Status Explanation
Description check ✅ Passed The pull request follows the required structure and clearly explains the problem, schedule changes, testing status, release status, documentation status, and agent context. The agent section includes …
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (1)
products/replay_vision/backend/temporal/schedule.py-146-152 (1)

146-152: 🚀 Performance & Scalability | 🟡 Minor | ⚡ Quick win

Do not trigger new schedules immediately.

trigger_immediately=True starts one workflow when Client.create_schedule creates the schedule. This bypasses the configured offset and jitter. The reconciler can create multiple scanner schedules concurrently, so their first workflows can read the same in-flight counts before staggering applies.

Set trigger_immediately=False in both schedule-creation branches and update the test expectation.

Suggested fix
-            client, schedule_id, schedule, trigger_immediately=True, search_attributes=search_attributes
+            client, schedule_id, schedule, trigger_immediately=False, search_attributes=search_attributes

Apply the same change in a_upsert_scanner_schedule, which has a separate creation branch.

-    assert create.call_args.kwargs["trigger_immediately"] is True
+    assert create.call_args.kwargs["trigger_immediately"] is False

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: PostHog/posthog/.coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: 59c08e91-98c4-4143-b1e9-e369adc488d5

📥 Commits

Reviewing files that changed from the base of the PR and between e107332 and 409d67b.

📒 Files selected for processing (8)
  • posthog/temporal/schedule.py
  • posthog/temporal/session_replay/surfacing_scoring_sweep/schedule.py
  • products/replay_vision/backend/temporal/estimates.py
  • products/replay_vision/backend/temporal/gemini_cleanup_sweep/schedule.py
  • products/replay_vision/backend/temporal/read_meter.py
  • products/replay_vision/backend/temporal/reconciler.py
  • products/replay_vision/backend/temporal/schedule.py
  • products/replay_vision/backend/temporal/search_suggestions.py

Included review availability: Your plan provides up to 12 included reviews per hour; 6 remain after this review.

@trunk-io

trunk-io Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge

View Full Report ↗︎ ⋅ Docs

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant