pull-request-dashboard

Pull Request Dashboard

A centralized shared workflow that builds and publishes a per-repository pull request triage dashboard for opted-in OpenTelemetry repositories. For each target repository, the workflow scans open PRs, classifies who has the next action, publishes the dashboard as a GitHub issue, and optionally sends Slack notifications.

The workflow runs from open-telemetry/shared-workflows and targets the repositories listed in repositories.json. Target repositories do not need to host any workflow files.

Webhook-triggered incremental runs keep active dashboards close to real time. Hourly backfill runs provide a backstop for missed or failed targeted refreshes.

The classification cache reuses prior results for unchanged review threads, minimizing Copilot token usage.

Data branch storage

The current dashboard state and delivery acknowledgements live on otelbot/pull-request-dashboard-state/* and otelbot/pull-request-dashboard-delivery/*. They must survive between runs, but their previous Git versions are not needed.

Data writers always replace each branch with a single parentless commit containing its complete current files. The Actions queue collector uses the same snapshot protocol. The cutover preserved stored JSON and delivery receipts; it did not migrate their formats.

The promoted stable processors and current canaries use unconditional snapshots. DATA_BRANCH_SNAPSHOTS is no longer required, and the obsolete repository variable can be deleted after the workflow environment cleanup is merged. See the storage and release requirements. Refresh schedules, notifications, and the stalled-run watchdog are unaffected.

Classification runtime

Classification uses the Python Copilot SDK with a task-specific system prompt and no tools or repository instructions. Each dashboard process reuses one client, but every batch of up to ten items gets a fresh session. This removes general coding-agent prompt overhead without sharing conversation history.

Running classifier evaluation locally

Use Python 3.11 or later and install the SDK and its matching runtime from the dashboard script directory:

cd .github/scripts/pull-request-dashboard
python -m pip install -r requirements.txt
python -m copilot download-runtime
python eval/score_reviewer_feedback.py --classifier reviewer_feedback --model gpt-5.6-luna

Set COPILOT_GITHUB_TOKEN to a token with Copilot access. Local runs also accept GH_TOKEN or GITHUB_TOKEN when no explicit Copilot token is set. Actions runs use the workflow token with copilot-requests: write, not the dashboard app's installation token.

The evaluation makes model calls and reports against the recorded cases without rewriting them. It runs at most four model calls concurrently, with fresh sessions and the same batching rules as production. Runtime telemetry is printed when the client closes. Cache identities include the system prompt and execution configuration; changing those settings cannot reuse results from the old configuration.

Webhook queue

Webhook events are coalesced by repository and pull request before they start GitHub Actions. One drain workflow claims a bounded batch and processes up to four repositories concurrently on one runner; pull requests from the same repository remain sequential. An event received while its pull request is being processed marks that item dirty and schedules one follow-up pass.

Webhook events, draft opens, and targeted PR/head refreshes use the live lane. Hourly/manual backfills and author-reminder write sweeps use an independent maintenance lane. Each lane has its own dispatcher lease and runner, so a maintenance batch cannot occupy live processing slots. Both use the same processor and keep ownership through calculation, accepted-state persistence, delivery, and issue publication. Failure retries the combined item. GitHub concurrency does not own the drain.

The lanes can calculate and persist classifications concurrently. Delivery and publication share a renewable per-repository lease, held only after calculation, to serialize comments, notifications, reminder writes, and dashboard publication.

An enqueue workflow succeeding means work was accepted, not completed. Its summary reports the item key, generation, and request ID. Run the dashboard workflow on main with request_item_key, request_generation, and request_id to inspect progress.

Stable repositories still use promoted release code; canaries use main. Backfills retain their bounded cursor, closed-PR cleanup, initial-population rules, and full-publication bookkeeping. Queue leases recover interrupted work, and hourly backfills remain the backstop for missed events.

The operator-controlled PR_DASHBOARD_EXECUTION_MODE must be owned or paused. Pausing accepts work but prevents new claims and successor dispatches. Incompatible ownership changes and manual writes outside the queue require the pause-and-retire procedure in WEBHOOK_SETUP.md. Removing retired legacy execution does not require another pause. There is no direct-dispatch fallback or return-to-legacy execution mode.

Dashboard columns

The dashboard groups open non-draft pull requests by who is expected to act next (e.g. Waiting on reviewers, Waiting on authors, Waiting on maintainers). Draft PRs are listed separately at the bottom unless large_repo rendering is enabled. Within each group, rows are sorted longest-waiting first. Every row has these six columns:

• PR — Pull request number and title, followed by any configured matching labels. The number autolinks to the PR on GitHub. Configured labels are rendered inline for both active and draft PRs.
• Author — Effective author of the PR, including in the draft section. For Copilot-authored PRs, this is the human assignee or first-commit committer when available.
• Reviewers — Reviewers who have engaged with the PR, excluding the effective author even when assigned or requested for review, each annotated with one or more icons:
  • ✅ active approval
  • ✔️ active approval (non-code-owner — does not count toward required_approvals)
  • 💬 unresolved feedback from reviewer
  • 📌 has tracked top-level feedback that still needs author action
  • 🔴 requested changes
  • ⏳ a review is in flight (a human reviewer other than the effective author was requested again after reviewing, or a Copilot review is pending while it holds the PR)
  • Icons combine when multiple states apply. For example, 💬📌 means the reviewer has unresolved inline feedback and top-level feedback that still needs author action; ✅ may accompany either or both.
  • Copilot appears only with unresolved inline feedback (💬) or a pending review that holds the PR (⏳). Completed reviews alone do not add Copilot to the column.
• CI — Aggregate check status across the PR's required status checks. Optional checks do not affect this column:
  • ✅ all required checks passing
  • ⏳ at least one required check pending, none failing
  • ❌ at least one required check failing
  • ? check data could not be fetched
• Conflicts — Whether the PR has merge conflicts against its base branch:
  • ✅ no conflicts
  • ❌ has conflicts
  • ? could not be determined
• Age — How long the PR has been waiting on the current next-action owner, not the calendar age of the PR. For example, a six-month-old PR that received a reviewer comment yesterday shows 1d when it is waiting on the author. When possible, this is based on the oldest pending thread for the routed party; otherwise it falls back to the latest activity from the opposite party, then to recent PR activity or PR creation time. Format: <1m, Xm, Xh, or Xd.

Merge conflicts remain visible without overriding normal CI, discussion, and approval routing. A conflicted PR can therefore stay with reviewers or maintainers when they can resolve it themselves. If missing required checks or a missing Copilot review hold the PR with its author, the live status names the conflict and the standard seven-day author reminder still applies. Missing automation does not expire into a dashboard delivery failure while the PR is conflicted.

How to opt in

Open a pull request that adds your repository to .github/scripts/pull-request-dashboard/repositories.json:

[
  {
    "name": "example-repo",
    "approver_teams": ["example-approvers"],
    "required_approvals": 1,
    "non_blocking_check_patterns": [
      "markdown-link-check / link-check",
      "codecov/*"
    ],
    "require_clean_copilot_review_branches": ["main"],
    "labels_to_display": ["size/*", "breaking change"],
    "slack_channel": "#example-maintainers",
    "slack_user_mapping": {
      "octocat": "U0123456789"
    }
  }
]

Fields:

Field	Required	Description
name	yes	Name of the repository under open-telemetry.
approver_teams	yes	GitHub team slugs whose members count as approvers.
required_approvals	no	Number of active approvals required for an open PR to be marked ready to merge. An approval stops counting while that reviewer has a pending re-review request. Defaults to 1.
labels_to_display	no	Case-sensitive shell-style label name patterns to display inline after PR titles. Exact names such as breaking change and wildcard patterns such as size/* are supported. Defaults to [], which displays no labels.
non_blocking_check_patterns	no	Check-name globs for non-required checks whose failures should be identified in the live PR status comment. When the PR is waiting on the author, matching failures are reported only when at least one required check is failing and are noted alongside those failures. On other routes, matching failures are shown separately. Matching checks remain informational and do not affect routing or the dashboard CI column.
require_clean_copilot_review_branches	no	List of base branch names for which a Copilot review of the current head is normally required before automatically routing a PR to reviewers or maintainers. Copilot findings use the same discussion rules as human feedback: a completed author reply resumes approval-based routing without requiring explicit thread resolution, while a commitment to more work keeps the action with the author. The live PR status links only threads needing author action. An effective /dashboard route:reviewers break-glass override bypasses this gate and every other routing blocker. The dashboard requests the first review on the first eligible refresh and re-requests review when a push has left the previous review stale. It does not duplicate a pending request. GitHub's quota-exhausted review notice does not count as a completed review, but it immediately releases the Copilot gate when no fallback token is configured or the fallback requester is exhausted; see the quota behavior below. A request counts as delivered only once GitHub confirms Copilot is a pending reviewer or finds a completed Copilot review of the current head. A gate that never reports holds the PR for at most four hours; after that the PR routes anyway, its status comment says which gate the dashboard stopped waiting for, and the run reports the stall. A conflict resets this clock because GitHub may not start checks or reviews until the conflict is resolved. PRs targeting unlisted branches are never gated. Defaults to [] (no branches gated).
slack_channel	no	Slack channel for notifications. Omit to skip Slack processing for this repository.
slack_user_mapping	no	Map of GitHub login to Slack user ID for at-mentions.
large_repo	no	If true, apply rendering presets that keep the dashboard body under GitHub's 65,536-character issue-body limit: cap each section (each Waiting on … table and the Draft pull requests table) at 100 rows, and omit the Draft pull requests section entirely. Truncated sections get a _More X PRs not shown_ footer. Defaults to false (no cap, drafts shown). Enable this for very large repos with hundreds of PRs.

labels_to_display only controls which labels are shown. It does not filter pull requests or affect dashboard routing, notifications, or status comments. All matching labels are displayed in the order returned by GitHub; a label matching more than one configured pattern is shown once.

For each listed branch, the dashboard requests initial Copilot reviews and re-reviews using its GitHub App installation token with pull-request write permission. An open, non-draft PR with no completed review is eligible on the first dashboard refresh where it would otherwise route to reviewers or maintainers. Conflicts, failing required checks, author-action feedback, and manual reviewer handoffs block requests; pending checks do not.

Copilot's "encountered an error and was unable to review" notice does not count as a completed review. The first error allows one retry on the next eligible refresh without requiring a new commit, unless a request is already pending. The error notice itself triggers a webhook refresh; hourly backfill is the backstop for a missed refresh. There is no additional retry delay.

After two error notices for the same commit, automatic requests stop and the Copilot gate releases immediately. The status comment explains why routing no longer waits for Copilot. Required checks and findings needing author action still apply. A new commit resets the error budget. The normal four-hour gate timeout remains the backstop for reviews that never report.

When Copilot reports quota exhaustion and no fallback token is configured, the dashboard immediately stops automatic review requests for that PR and releases the Copilot review gate without waiting for a completed review or the four-hour timeout. Only the Copilot gate is released: required-check gates and actionable feedback, including Copilot threads needing author action, still apply.

Maintainers can optionally configure the COPILOT_REVIEW_FALLBACK_TOKEN workflow secret with a token allowed to request Copilot reviews. The owned drain passes it to the selected processor for both channels and both lanes. After the first quota notice, the dashboard switches review requests to this fallback requester and keeps the Copilot gate active under its normal rules. Existing quota notices do not exhaust the fallback; a new quota notice after the fallback attempt does. Once the fallback is exhausted, automatic requests stop and the Copilot gate is immediately released for that PR, including after later pushes. Required checks and actionable feedback are not bypassed in either case.

Ask a maintainer or admin to add the repository under Repository access.

Once the PR is merged, the dashboard will pick up your repository on its next hourly backfill run. To run it sooner, see Manual backfill run. The dashboard issue is discovered dynamically in your repository by the dashboard label and Pull Request Dashboard title; if it does not exist, the publish step creates the label and issue.

Review feedback lifecycle

Inline review threads and top-level feedback have different lifecycles on GitHub. Top-level feedback means a standalone PR comment or submitted review summary that is not attached to an inline review thread.

• An inline thread remains on the dashboard until someone marks the conversation resolved on GitHub or GitHub marks its code anchor outdated. A completed author reply hands the action away from the author but does not close the thread. If the PR has enough approvals, it resumes approval-based routing and can move to maintainers without reviewer confirmation; otherwise it waits on reviewers. A reply that commits to more work keeps the PR with the author. Author-only inline threads are treated as annotations rather than review feedback unless a non-author joins them. These rules apply equally to human and Copilot feedback.
• Top-level feedback has no resolved state. The dashboard therefore tracks each actionable top-level feedback item independently. 📌 means that one or more of those items are waiting on the author.
• Reviewer badges reflect GitHub's latest review state. A CHANGES_REQUESTED state affects only the reviewer's badge; it does not affect dashboard classification or routing. 🔴 remains until a later approval or dismissal clears that state. Empty review summaries are ignored; any inline comments are tracked through their own threads.
• When a human reviewer is requested again, ⏳ replaces their previous approval badge and that approval stops counting toward required_approvals. The PR waits on reviewers unless its other active approvals meet the threshold. This is intentionally stricter than GitHub, which can continue to accept the old approval when the repository does not dismiss stale approvals. 🔴 stays alongside ⏳ while the changes-requested review blocks the merge. A later approval or dismissal clears 🔴, but ⏳ remains until the requested re-review arrives.

Closing top-level feedback

An explicit author reply is the only thing that closes an actionable top-level feedback item: a later standalone PR comment by the author that answers, decides, clarifies, or reports what was done. The reply lets authors explain why a suggestion was not applied, ask a clarifying question, or otherwise close the dashboard action. If the author instead commits to future work in the current PR, such as testing or making another change later, the item remains waiting on the author. The dashboard intentionally treats a reply as a handoff signal, not proof that the reviewer agrees with the outcome.

Routing after a reply

Item	Before an author reply	After an author reply	When it clears
Inline author action	Waiting on the author	A completed reply resumes approval-based routing; a commitment to more work remains with the author	The conversation is resolved or its code anchor becomes outdated
Top-level author action	Waiting on the author; 📌 is visible	Addressed; 📌 disappears and normal approval-based routing resumes	Immediately after an explicit author reply

GitHub remains responsible for enforcing blocking review states when a maintainer attempts to merge.

This can route an approved PR to maintainers before a reviewer confirms the author's response. That handoff is deliberate: the open-thread badge remains visible so maintainers can merge or defer to the reviewer, while the dashboard avoids leaving an author marked as blocked after completing their part.

Live PR status comment

After the first full dashboard run has populated repository state, each targeted PR update creates or updates one dashboard-managed status comment on that PR. The comment presents the time its status was last refreshed in UTC and a compact Waiting on and Next step summary. The timestamp is refreshed on every targeted PR update, even when the dashboard status is unchanged. When the author has the next action, pending review feedback is summarized as a count and its inline-thread and top-level-feedback links are nested under that next step, followed by guidance for giving each item a clear outcome. When failing required checks and feedback both need author action, the two are listed under Next steps. At most 20 feedback links are shown across both groups; when more exist, a note reports how many of the total are shown. Draft PRs show that they are waiting for the author to move the PR out of draft to request review.

A failing required status check routes a human-authored PR to the author ahead of review and approval state. The live comment calls out required CI failures explicitly and combines that reason with review feedback when both need author action. When a repository configures non_blocking_check_patterns, matching failed checks are named in a note alongside the required-check action when the PR is waiting on the author because at least one required check is failing. On other routes, matching failures are shown separately. Optional check failures do not affect routing. When the base branch ruleset requires code scanning results, the check GitHub publishes for each configured tool is treated as required, including its neutral "could not determine the alerts introduced by this pull request" outcome, which holds the merge even though GitHub does not mark that check as required. PRs from unattended automation do not route to their author because the automation cannot act on a dashboard request. A Copilot-delegated PR can route to a human author recovered from an assignee or the first commit's committer. Known maintenance bots still route to maintainers after one approval.

A hidden marker lets the workflow update the comment in place and upgrade existing one-time guidance comments rather than creating duplicates. Status comments are refreshed automatically when their dashboard status or format changes, including on inactive pull requests. Clear outcomes keep stale or ambiguous feedback from being routed to the wrong person.

Reviewers should prefer inline comments for feedback requiring explicit resolution. See RATIONALE.md for the tradeoffs behind this behavior.

Targeted updates received before the first full dashboard run are ignored.

Reviewer routing override

When the dashboard says a pull request is waiting on its author, the author can comment /dashboard route:reviewers to request human help. The command is a break-glass handoff. It forces Waiting on reviewers regardless of review feedback, approvals, required checks, Copilot review, or merge conflicts. It is useful when the author is stuck or needs a reviewer or maintainer to explain what to do next. Members of the repository's approver_teams can use the same command.

The dashboard binds an authorized command to the head it sees when it first reads that command, and records that head in its acknowledgement reply. The handoff stays active while the pull request head matches the recorded one, so a later push restores normal routing and gates. Actionable human reviewer feedback posted after the command timestamp also ends the handoff, so normal routing and gates apply again. Praise, informational comments, bot feedback, classification failures, and feedback posted at or before the command timestamp do not end it. A push between the command and the pass that reads it is part of the same handoff, because the dashboard binds to the head it can actually see.

The dashboard replies to an unauthorized command explaining that only the author or an approver can use it, and replies to any unrecognized /dashboard command.

Author reminder

The dashboard posts one reminder when a pull request remains in Waiting on authors for one week. The friendly reminder @-mentions the author, links to the dashboard-managed status comment containing the current blockers, says that answering the open items resumes normal approval-based routing, and advertises /dashboard route:reviewers as a break-glass handoff for any other reason, including the dashboard routing the pull request wrongly. The same one-week clock applies while the author route is held on required checks, Copilot review, or merge conflicts.

When the dashboard routes the pull request to approvers or maintainers, it appends an italic note saying that the pull request is no longer waiting on the author, then marks the comment Outdated so GitHub collapses it. The original reminder remains available when the comment is expanded. If a temporary gate hold, episode reset, or removal from the dashboard ends the author-waiting episode, the appended note says only that the reminder no longer reflects the current dashboard state. Both notes link to the live status without naming the next route, which may change again. Leaving Waiting on authors also resets the one-week clock. If the pull request later returns to Waiting on authors and remains there for another week, the dashboard posts another reminder. Reminders are delivered by hourly runs when the pull request is next refreshed, so a due reminder in a large repository may wait for a later round-robin run.

Configuration

The dashboard separates calculation from delivery. The update job uses repository-scoped, read-only GitHub App access to calculate routing and persist pending work. A serialized publishing job holds pull request and issue write access and durably delivers status comments, author reminders, Copilot re-review requests, Slack notifications, and the dashboard issue. Successful delivery receipts live on a separate delivery branch. Each backfill records a full-publish generation on the state branch. If GitHub cancels its pending publisher, the next publisher for that repository performs a full delivery and acknowledges the generation after publishing the issue. Otherwise, webhook-triggered publishers continue to deliver status comments and Slack only for their triggering PR. Subsequent publishers, including hourly runs, retry outstanding full deliveries. Hourly failure reporting tolerates cancellations but reports failed jobs. Pending generations are not separately monitored for age.

Each repository can route Slack notifications to its own slack_channel and map GitHub logins to Slack user IDs via slack_user_mapping. Repositories without slack_channel configured do not send Slack notifications.

Staged rollout

Dashboard changes reach repositories in two stages, so a regression shows up on a small set of repositories before it reaches everyone.

• Canary repositories run processor scripts from the drain's immutable triggering commit and pick up a change as soon as it merges. The canary set is the CANARY_REPOSITORIES list in pull-request-dashboard-drain.yml, which currently holds shared-workflows itself and opentelemetry-java-instrumentation.
• Every other repository runs them from the promoted release's commit SHA in stable-release.json. The file records one lowercase 40-character SHA and one stable semantic release.

Both channels always use the webhook queue. Release selection changes processor code, not webhook routing or execution ownership.

Repository configuration is never staged. repositories.json is always read from the commit that triggered the run, so opting a repository in, or changing its settings, takes effect on the next run in both channels.

Promoting a change

1. Merge the change. Canary repositories pick it up on their next drain.
2. Let it soak.
3. Cut a release with the Release workflow.
4. Run the Promote pull request dashboard workflow from main and enter the release tag. It accepts only a newer, published, non-prerelease version and opens a pull request updating the SHA and release in stable-release.json. Both lanes load stable processor scripts and dependencies from that immutable commit.
5. Review and merge the generated pull request to promote the release. The merge selects the release for subsequent stable drains without redeploying the bridge.

The release configuration is not an action dependency and is not managed by Renovate. Readers and the promotion helper reject mutable refs and malformed releases. The promotion workflow uses the repo-specific shared-workflows OTELBOT App to push the branch and open the pull request so its normal checks run.

Rolling back

Revert the promotion pull request to an earlier release commit. If the reverted change bumped no delivery version, the stable channel resumes normally. If it did, those repositories stop delivering rather than repeat reminders they can no longer see as sent, so recovery is to roll forward with a fix. Canary repositories are unaffected either way, since they never run pinned code.

Changing the workflow

The owned worker loads stable scripts without replacing its own checkout. Processor CLI changes must remain compatible with the pinned stable release. repositories.json changes must be additive for one promotion cycle, because pinned code reads live configuration. Renames and removals land after the code that understands them has been promoted.

Prerequisites

The target repository GitHub App must be installed on your repository. Follow the repository-access step under How to opt in.

Manual backfill run

To manually run a dashboard backfill, open the Pull request dashboard workflow, choose Run workflow, and populate the repository field with the target repository name under open-telemetry, for example opentelemetry-java-instrumentation. Leave repository empty to backfill every configured repository. Each run processes at most 50 PRs per repository; run it again to continue through larger repositories.