[ChickenBenny]

Tracking issue

Closes #8193 (follow-up to #8200)

Why are the changes needed?

lib/pq sends a nil on Listener.Notify after it reconnects, because NOTIFYs sent while it was down are lost (docs). processNotifications skipped that nil, so a watcher whose run or action changed during a Postgres outage never heard about it. The console and run.wait() stayed on the old phase with no error.

Postgres doesn't replay NOTIFYs, but nothing needs them: the DB already has the new state, and each watcher knows what it is watching.

What changes were proposed in this pull request?

Three commits, easier to review separately:

1. Refactor, no behavior change. Subscribers become a subscription struct instead of a bare channel, and the duplicated register/unregister code of the four Watch* functions moves into subscribe / unsubscribe.

2. Single-key watchers. Each subscription gets a one-slot resync channel. On nil, processNotifications signals every subscriber without blocking; repeated signals coalesce. WatchActionUpdates and WatchRunUpdates re-read their action or run from the DB. This covers run.wait() / WatchActionDetails, WatchClusterEvents and WatchRunDetails.

3. List watchers. WatchAllRunUpdates and WatchAllActionUpdates pass a resync on as a nil update (documented on the interface), and the service re-lists:

   • WatchRuns (console run list) re-runs its initial list query with the same filter.
   • WatchActions (console action tree) re-runs listAndSendAllActions, skipping actions whose updated_at the stream has already seen. Without that, upserting an existing action marks it and its ancestors as changed, and the whole tree would be sent again.

   A failed re-list is logged and the stream stays open, the same as other read errors in these streams.

Notes for review:

• Load on reconnect: every watcher on the replica re-reads at once. Reconnects are rare, and the single-key reads are primary-key lookups. The list re-reads are the same queries the streams already run when they open.
• Duplicates: a single-key resync sends the current state even if nothing changed; those streams just send the latest state. WatchActions sends only what changed.
• Dropped notifications: a full subscriber buffer still drops notifications, as before. Resyncing on overflow would let a busy run on the same replica trigger re-lists in every other stream, so I left it out.
• WatchRuns limit: the re-list uses the stream's initial query, including its limit of 100, so it is as complete as the snapshot a fresh stream gets.

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

• fix(runs): keep LISTEN/NOTIFY reader alive when listener ping fails #8200: keeps the listener goroutine alive when Ping fails (part 1 of [BUG] [V2] Run/action updates stop after a brief Postgres outage: runs-service listener exits on a failed Ping #8193)

cc @AdilFayyaz