Skip to content

feat(canvas): name the selected block when asking the agent - #107793

Open
puemos wants to merge 2 commits into
masterfrom
posthog/canvas-ask-agent-block-reference
Open

puemos wants to merge 2 commits into
masterfrom
posthog/canvas-ask-agent-block-reference

Conversation

@puemos

@puemos puemos commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

Problem

  • On a canvas with several similar blocks, "Ask the agent about this" does not say which block the person selected.
  • The prefilled message names only the file, for example Change the channel kpi card at src/canvas.tsx: .
  • In blind tests with that message, the agent changed all six KPI cards, not the one the person selected.

Changes

  • The message now names one block. The first signal that applies wins:
    • The blockId, when the block has one.
    • Otherwise, the block's opening tag from the source, for example Change only this channel kpi card `<ChannelKpiCard channel="Referral" goal={800} />` in src/canvas.tsx: .
    • "match 2 of 2 for this code in the file" when the same tag occurs more than once.
    • When one piece of code renders many times on the page (a .map(), or markup inside a shared component), the page position, the props, and the visible text.
  • The button shows a spinner until the editor saves. The message then matches the source the agent reads. After a failed save, the button works again.
  • Line numbers and CSS selectors are left out on purpose. In blind tests, both picked the wrong card after the page and the source got out of sync.
  • The building-canvases skill now asks for a blockId on each editable() use, so new canvases have stable ids.
  • The editor runtime reports two new selection fields, instance and visibleText. The message itself is built in @posthog/core (blockReference.ts).

Not rendered: the only visual change is the spinner on the button, and no screenshot was taken.

How did you test this code?

  • blockReference.test.ts (5 cases): catches a message that loses the block id, cuts the opening tag at an => inside braces, numbers duplicate tags wrong, or drops the page position and text for code that renders many times.
  • Canvas tests in @posthog/ui and @posthog/core, typecheck for both packages, Biome, and hogli lint:skills pass locally.
  • Blind agent test on 8 invented cases, with messages from the real builder: literal cards, .map() cards, a title inside a shared component, the agent's source copy out of date, duplicate tags, a bare heading, and blockId with and without an out-of-date copy. Sonnet found the correct block in 8 of 8. Haiku found it in 8 of 8, but sometimes planned the change on the shared component.
  • Not checked: the running desktop app.

Release status

  • No feature flag controls this change
  • This change is behind a feature flag and is not available to users
  • This change makes a previously flagged feature available to everyone

Automatic notifications

  • Publish to changelog?

Docs update

None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: PostHog Desktop (Claude Code), claude-opus-5-5

  • Skills invoked: /writing-tests, /writing-user-facing-copy, /writing-skills, /writing-pr-descriptions.
  • Follows #107763, which merged before this work was pushed, so it is a new PR.
  • Earlier rounds tested line numbers, snippets, props alone, and CSS selectors. Sonnet and Haiku subagents played the canvas agent, on invented canvases.
  • "Change only this" replaced "Change the" after Haiku read the old wording as a change to the component.
  • The source was internal product feedback. The PR contains no material from it.

Created with PostHog Desktop

🤖 Generated with Claude Code

"Ask the agent about this" sent only the file name, so the agent could not tell which of several similar blocks the person selected. The prompt now names the block by blockId, or by its opening tag in the source, with the page position, props, and visible text when one piece of code renders many times. The button waits for the editor to save, so the prompt matches the source the agent reads. The canvas skill asks for a blockId on each editable() use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Generated-By: PostHog Desktop
Task-Id: 2741091a-93e9-4bc1-965a-c9ddc535ef67
@puemos puemos self-assigned this Sep 28, 2026
@trunk-io

trunk-io Bot commented Sep 28, 2026

Copy link
Copy Markdown

Merging to master in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@github-actions github-actions Bot added the feature/desktop Feature Tag: Desktop label Sep 28, 2026
@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

React Doctor found 1 issue in 1 file · 1 warning.

1 warning

packages/ui/src/features/canvas/blocks/canvasEditRuntime.ts

Reviewed by React Doctor for commit 63373f4.

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

✅ Complexity (TypeScript) — clean

Cyclomatic complexity above the limit in changed typescript files (10 for production files, 15 for test files). Warn only: worth simplifying when you next touch these functions.

✅ Duplication (Python) — clean

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

✅ Duplication (TypeScript) — clean

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

@hosthog

hosthog Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

HostHog preview — posthog-desktop-web

Latest build (63373f4): https://f0c4cc444da24d8799bb09609a059f36.hosthog.dev

Employee-gated; every push gets a fresh URL whose content never changes. All previews stop serving when the PR closes.

@coderabbitai

coderabbitai Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

📝 Walkthrough

Walkthrough

Canvas selections now carry normalized visible text and rendered-instance details for repeated source elements. A new prompt builder formats references using block IDs or available source and selection context, with tests for its output. The canvas panel now uses this builder for agent prompts and derives a waitingForSave value. Canvas-building guidance now recommends unique blockId values for component uses.

Priority: ➖ Normal

Merge Risk: 🔵 Low · up to 63373

The change is mergeable with a small documentation correction: add a blockId to the usage example so copied code follows the guidance. The opening-tag escape concern is fixed.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 63373

The change provides more precise block descriptions without automatically starting an edit. Save failures or conflicts can leave those descriptions inconsistent with saved content, weakening the intended single-block scope. No privilege expansion was established, but later execution permissions were not fully assessed.

Retained concerns

  • Low · reliability · inferred: After a save failure or version conflict, the selected-block action can generate tag and occurrence references from unsaved local files. Those references may not identify the same block in the live source subsequently read for editing, weakening intended single-block failure containment. Normal save waiting, stable-ID references, the existing save warning, and editable composer review reduce the risk; no automatic wrong-block modification or authorization bypass was demonstrated.
Security review details

Security Blast Radius

  • inferred — The directly observed effect is one selected non-root block's description in the current canvas composer. The formatter itself cannot mutate source or invoke tools. Later editing authority is inherited from the existing submitted-task flow, so the description must not be treated as proof that execution is restricted to that block or canvas.

Security Findings and Attack Paths

  • inferred — Content controlled through the selected canvas can now travel from DOM text or props into an editable user instruction and, after submission, the editing flow. This establishes an expanded content path, not a verified instruction-injection exploit. User review is an intervening control; downstream instruction handling and tool authority remain unverified.

Trust Boundaries and Controls

  • observed — The runtime escapes the source attribute used in its DOM selector and normalizes captured visible text. The formatter rejects unsafe IDs from its direct-ID branch, but interpolates included text and string props inside quotation marks without escaping embedded quotation marks. The consumer receives a plain string, not a separate trusted instruction and untrusted context structure.

Resilience and Maintainability Implications

  • inferred — The new save waiting improves normal-path consistency but does not bind a generated description to a persisted revision. Failure or conflict releases waiting, and the eventual submission carries the instruction rather than the local source snapshot. The resulting recovery-state mismatch threatens edit containment, not an established tenant or permission boundary.

Hardening Proposals

  • proposed — Preserve rendered text, props, and source snippets as explicitly untrusted descriptive context rather than relying on quotation marks or length limits to distinguish them from user instructions.
  • proposed — For save-error and conflict recovery, distinguish unsaved local descriptions from persisted references and revalidate identity against the revision consumed at submission. Keep recovery available without presenting unsaved tag or occurrence details as authoritative live-source identity.
🚥 Pre-merge checks | ✅ 1
✅ Passed checks (1 passed)
Check name Status Explanation
Description check ✅ Passed The description is complete and matches the required template. It explains the user problem, observable changes, testing performed, release status, documentation status, agent context, and known limit…
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (1)
products/desktop/packages/core/src/canvas/blockLibrary/blockReference.ts-36-36 (1)

36-36: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Handle an even number of backslashes before a closing quote.

If a JSX attribute value ends in an escaped backslash, Line 36 treats its closing quote as escaped. openingTag can then include content beyond the opening tag, so the agent receives the wrong source reference. Track whether the preceding backslash run has odd length before treating a quote as escaped.


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: PostHog/posthog/.coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: fccef08e-4e0e-4bfe-98aa-f5533d68ae47

📥 Commits

Reviewing files that changed from the base of the PR and between 7d5de91 and f521c64.

📒 Files selected for processing (6)
  • products/canvas/skills/building-canvases/SKILL.md
  • products/desktop/packages/core/src/canvas/blockLibrary/blockReference.test.ts
  • products/desktop/packages/core/src/canvas/blockLibrary/blockReference.ts
  • products/desktop/packages/ui/src/features/canvas/blocks/CanvasBlocksPanel.tsx
  • products/desktop/packages/ui/src/features/canvas/blocks/canvasEditRuntime.ts
  • products/desktop/packages/ui/src/features/canvas/blocks/canvasSourceStore.ts

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

Comment thread products/canvas/skills/building-canvases/SKILL.md Outdated
Comment thread products/desktop/packages/core/src/canvas/blockLibrary/blockReference.ts Outdated
@puemos puemos added the stamphog Request AI approval (no full review) label Sep 28, 2026
@puemos
puemos marked this pull request as ready for review September 28, 2026 15:18
@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team September 28, 2026 15:19
@pr-assigner-resolver-posthog

Copy link
Copy Markdown

👀 Auto-assigned reviewers

These soft owners were skipped because they only have minor changes here. Nothing blocks merge, so self-assign if you'd like a look:

  • @PostHog/team-posthog-desktop (products/canvas/product.yaml)

Soft owners come from each directory's owners.yaml and each product's product.yaml (resolved nearest-file-wins). The locator after each owner is the file that decided it. Generated files and lockfiles are ignored when deciding ownership.

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved — this change needs a human reviewer.

Re-add the stamphog label to request another review once you have addressed this.

CodeRabbit flagged an unresolved Major correctness issue that I confirmed by reading the code: when a selected block has a blockId, blockReferencePrompt returns early and drops the instance/props details entirely, even though the SKILL.md guidance itself recommends dynamic per-item blockIds (e.g. a template literal built from a list key) that won't appear literally in the source — which is exactly the ambiguous-block scenario this PR exists to fix. This is a real, unaddressed functional gap in the core feature, not a nitpick.

  • Author wrote 100% of the modified lines and has 2 merged PRs in these paths (familiarity STRONG).
  • coderabbitai[bot] reviewed the current head.
  • 5 of the 6 changed files are governed by products/desktop/AGENT_APPROVALS.md.
  • Unresolved CodeRabbit 'Major' comment on blockReference.ts: when blockId is set, the function short-circuits and omits instance/props/visibleText, so a repeated block with a computed (non-literal) blockId still gives the agent an ambiguous reference — undermining the PR's stated goal.
  • Unresolved CodeRabbit 'Minor' comment on blockReference.ts line 36: quote-escaping check in openingTag doesn't handle an even number of preceding backslashes, so it can occasionally include extra source beyond the real opening tag (lower severity, quick fix).
Gate mechanics and policy version
Gate Result
prerequisites ✓ all clear
deny-list ✓ no deny categories matched
size ✓ 153L, 4F substantive, 252L/6F incl. docs/generated/snapshots — within ceiling
tier ✓ T1-agent / T1c-medium (252L, 6F, single-area, feat)
stamphog 2.2.0 .stamphog/policy.yml @ f521c64 · reviewed head f521c64

@stamphog stamphog Bot removed the stamphog Request AI approval (no full review) label Sep 28, 2026
const { label, source, blockId, instance } = input;
const where = source ? ` in ${source.file}` : "";
if (blockId)
return `Change only this ${label} with blockId ${blockId}${where}: `;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low: Prompt injection through block references

blockId is copied verbatim into the task's first user message, and the fallback path similarly includes runtime props and visible text. Because the skill now recommends deriving IDs from item.id, an attacker who supplies a queried analytics value can embed instructions that run when an editor submits this prefilled prompt; canvas tasks use bypass-permissions mode with full PostHog MCP scopes. Use an opaque, validated identifier containing only safe characters, avoid placing rendered data in the instruction, and pass any necessary display metadata as explicitly untrusted structured context to a least-privileged task.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partly fixed in 63373f4: the prompt uses a blockId only when it matches ^[\w.:-]{1,80}$. Other IDs fall back to the source reference. Props and visible text stay in the draft, clipped to short lengths, because they are the only way to find a repeated item. The person sees and edits the draft before they send it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detail. I can't automatically confirm this is safe to dismiss, so I'm leaving the thread open for a maintainer to make the call.

@veria-ai

veria-ai Bot commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

PR overview

This pull request updates canvas agent requests to identify the currently selected block, using block references and fallback block details in the initial prompt.

One prompt-injection path remains open because attacker-controlled block identifiers or rendered values can be inserted directly into an agent prompt. If an editor submits that prompt, injected instructions could execute through an agent running with broad PostHog permissions; no issues have yet been addressed.

Open issues (1)

Fixed/addressed: 0 · PR risk: 7/10

Append the page instance, props, and visible text to a blockId reference, ignore block ids with unsafe characters, handle escaped backslashes before a closing quote, and add blockId to the SignupFunnel example props.

Generated-By: PostHog Desktop
Task-Id: 5bd38eba-eb1f-4af7-86c2-bc12b26fb6d7

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (1)
products/canvas/skills/building-canvases/SKILL.md-85-85 (1)

85-85: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Add blockId to the usage example.

Line 84 shows a SignupFunnel use without blockId, but this step requires one for every use. Copying the example omits data-ph-block-id, so the component cannot use the block-ID reference path. Add a blockId to the example.


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: PostHog/posthog/.coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: 38ad638a-7559-4c40-ae04-8cc03bd20120

📥 Commits

Reviewing files that changed from the base of the PR and between f521c64 and 63373f4.

📒 Files selected for processing (3)
  • products/canvas/skills/building-canvases/SKILL.md
  • products/desktop/packages/core/src/canvas/blockLibrary/blockReference.test.ts
  • products/desktop/packages/core/src/canvas/blockLibrary/blockReference.ts

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 7 remain after this review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature/desktop Feature Tag: Desktop

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants