chore(posthog-ai): move exec command parsing out of the eval harness

[rafaeelaudibert]

Problem

• Production code that needs to read a PostHog AI tool call would have to import the eval harness.
• Layer 3 of 9 in the stack that replaces feat(growth): suggest a scout or notebook after a PostHog AI turn #101991. The turn suggestion transcript in layer 5 reads tool calls this way.

Changes

A mechanical move. Nothing is user-visible.

• parse_exec_command, normalize_tool_name and INFO_SYNTHETIC_PREFIX move to products/posthog_ai/backend/exec_commands.py.
• The eval harness and the eval scorers import them from the new module.
• Their tests move with them.

How did you test this code?

• No new behavior, so no new tests. The moved tests cover the parser in its new home.

Release status

• No feature flag controls this change
• This change is behind a feature flag and is not available to users
• This change makes a previously flagged feature available to everyone

Automatic notifications

• Publish to changelog?

Docs update

None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5.

• Skills invoked: /stacking-prs, /writing-pr-descriptions.
• CodeRabbit CLI pass skipped by the user's standing choice.

🤖 Generated with Claude Code

[github-actions]

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

✅ Duplication (Python) — clean

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

✅ Duplication (TypeScript) — clean

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

⚠️ Backend coverage — 91.0% of changed backend lines covered — 31 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ██████████████████░░ 91.0% (325 / 356)

File	Patch	Uncovered changed lines
products/tasks/backend/facade/task_run_signals.py	75.0%	25
products/tasks/backend/facade/api.py	83.5%	709–710, 725, 727–732, 734–738, 4542–4544, 4549, 4552–4553, 4593, 4640
products/posthog_ai/backend/exec_commands.py	87.2%	51, 60, 69, 73, 81–82
products/tasks/backend/models.py	92.6%	981, 985

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 146989594731823 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
platform_features	██░░░░░░░░░░░░░░░░░░ 12.1%	7 / 58
warehouse_sources_queue	██████░░░░░░░░░░░░░░ 29.1%	92 / 316
demo	████████████░░░░░░░░ 57.8%	1,545 / 2,673
data_tools	████████████░░░░░░░░ 61.2%	90 / 147
aeo	██████████████░░░░░░ 70.5%	467 / 662
ai_gateway	███████████████░░░░░ 75.0%	9 / 12
batch_exports	████████████████░░░░ 81.2%	21,454 / 26,424
apm	█████████████████░░░ 84.1%	1,306 / 1,553
ml_inference	█████████████████░░░ 86.7%	451 / 520
cdp	██████████████████░░ 88.2%	4,548 / 5,155
mcp_analytics	██████████████████░░ 88.6%	4,704 / 5,308
signals	██████████████████░░ 88.7%	52,403 / 59,082
product_tours	██████████████████░░ 89.3%	1,331 / 1,491
dashboards	██████████████████░░ 89.8%	7,234 / 8,056
data_warehouse	██████████████████░░ 89.9%	13,738 / 15,285
cohorts	██████████████████░░ 90.2%	8,238 / 9,138
notebooks	██████████████████░░ 90.2%	15,287 / 16,945
streamlit_apps	██████████████████░░ 90.7%	2,625 / 2,895
managed_warehouse	██████████████████░░ 90.9%	10,215 / 11,234
tasks	██████████████████░░ 91.1%	73,926 / 81,124
data_modeling	██████████████████░░ 91.4%	10,491 / 11,483
business_knowledge	██████████████████░░ 91.6%	6,899 / 7,528
engineering_analytics	██████████████████░░ 91.7%	11,002 / 11,999
exports	██████████████████░░ 91.8%	9,685 / 10,555
ai_training	██████████████████░░ 92.2%	356 / 386
conversations	███████████████████░ 92.5%	28,705 / 31,025
early_access_features	███████████████████░ 92.6%	1,341 / 1,448
managed_migrations	███████████████████░ 92.7%	1,581 / 1,705
visual_review	███████████████████░ 92.8%	9,244 / 9,966
canvas	███████████████████░ 92.8%	6,877 / 7,409
approvals	███████████████████░ 93.0%	3,919 / 4,214
mcp_registry	███████████████████░ 93.1%	1,670 / 1,794
error_tracking	███████████████████░ 93.1%	15,708 / 16,874
notifications	███████████████████░ 93.2%	1,145 / 1,229
slack_app	███████████████████░ 93.2%	13,677 / 14,674
surveys	███████████████████░ 93.3%	6,571 / 7,040
stamphog	███████████████████░ 93.4%	7,716 / 8,262
context_layer	███████████████████░ 93.8%	3,373 / 3,595
web_analytics	███████████████████░ 93.9%	21,653 / 23,051
ai_observability	███████████████████░ 94.0%	20,588 / 21,892
alerts	███████████████████░ 94.1%	8,553 / 9,094
billing_alerts	███████████████████░ 94.1%	2,094 / 2,226
mcp_store	███████████████████░ 94.4%	8,940 / 9,472
wizard	███████████████████░ 94.4%	5,791 / 6,134
reminders	███████████████████░ 94.8%	760 / 802
workflows	███████████████████░ 94.8%	13,438 / 14,170
review_hog	███████████████████░ 94.9%	11,490 / 12,109
annotations	███████████████████░ 95.1%	817 / 859
customer_analytics	███████████████████░ 95.1%	24,669 / 25,932
endpoints	███████████████████░ 95.1%	9,211 / 9,681
legal_documents	███████████████████░ 95.2%	2,311 / 2,427
marketing_analytics	███████████████████░ 95.3%	19,047 / 19,991
posthog_ai	███████████████████░ 95.4%	2,489 / 2,610
logs	███████████████████░ 95.4%	15,235 / 15,967
tracing	███████████████████░ 95.4%	3,483 / 3,650
growth	███████████████████░ 95.4%	9,812 / 10,282
actions	███████████████████░ 95.5%	756 / 792
messaging	███████████████████░ 95.9%	3,766 / 3,927
skills	███████████████████░ 95.9%	6,649 / 6,932
replay_vision	███████████████████░ 96.0%	26,750 / 27,878
autoresearch	███████████████████░ 96.0%	7,716 / 8,037
product_analytics	███████████████████░ 96.2%	28,495 / 29,617
revenue_analytics	███████████████████░ 96.4%	1,876 / 1,946
access_control	███████████████████░ 96.4%	7,122 / 7,386
user_interviews	███████████████████░ 96.5%	2,859 / 2,963
feature_flags	███████████████████░ 96.5%	25,146 / 26,046
experiments	███████████████████░ 96.6%	33,963 / 35,146
warehouse_sources	███████████████████░ 97.2%	443,966 / 456,678
data_quality	████████████████████ 97.7%	7,592 / 7,774
links	████████████████████ 97.9%	234 / 239
security	████████████████████ 98.1%	1,258 / 1,283
metrics	████████████████████ 98.1%	4,085 / 4,166
analytics_platform	████████████████████ 98.3%	2,778 / 2,827
data_catalog	████████████████████ 98.3%	3,932 / 3,999
pulse	████████████████████ 98.5%	2,043 / 2,075
live_debugger	████████████████████ 99.2%	626 / 631
field_notes	████████████████████ 99.4%	172 / 173

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

[greptile-apps]

[Medium risk] Moves command parsing logic to a shared backend module.

The PR appears safe to merge.

_{Reviews (1) · Last reviewed commit: "refactor(posthog-ai): move exec command ..."}

[stamphog]

Approved.

Pure mechanical code move (parsing helpers relocated from the eval harness into a shared backend module) with no behavior change and updated imports/tests; not risky territory, and Greptile's review found it safe.

• 👍 on the PR from greptile-apps[bot].

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	198L, 8F substantive, 236L/10F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1c-medium (236L, 10F, single-area, chore)
stamphog 2.1.0		.stamphog/policy.yml @ dd2b229 · reviewed head dd2b229

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: PostHog/posthog/.coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: 20e73a6d-5a40-493a-8f90-a32f34ab40cf

📥 Commits

Reviewing files that changed from the base of the PR and between 0a9b57d and dd2b229.

📒 Files selected for processing (10)

• products/posthog_ai/backend/exec_commands.py
• products/posthog_ai/backend/tests/test_exec_commands.py
• products/posthog_ai/eval_harness/log_parser.py
• products/posthog_ai/eval_harness/scorers/__init__.py
• products/posthog_ai/eval_harness/test/test_log_parser.py
• products/posthog_ai/evals/cli_mcp/scorers.py
• products/posthog_ai/evals/cli_mcp/skill_distribution_scorers.py
• products/posthog_ai/evals/cli_mcp/skill_usage_scorers.py
• products/posthog_ai/evals/product_analytics/scorers.py
• products/posthog_ai/evals/retrieval/scorers.py

Included review availability: Your plan provides up to 12 included reviews per hour; 4 remain after this review.

---

📝 Walkthrough

Walkthrough

The PR adds normalize_tool_name and parse_exec_command to a backend module. The log parser uses the shared parser instead of its local implementation. Evaluation tests and scorer modules update imports to use the shared definitions. New unit tests cover tool-name normalization.

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to dd2b2

The parser move has no identified behavior regression and is ready for normal merge checks.

Security Architecture Review

Security architecture risk: 🔵 Low · up to dd2b2

The shared parser changes where evaluation code gets command names, but the inspected callers only read recorded tool calls. No new command-execution path was identified. Coverage of future production use remains limited.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The inspected reach of the shared helper is evaluation parsing and scoring. Its new location makes later production reuse possible, but no production execution caller was identified in this change.

Security Findings and Attack Paths

• inferred — No new path from a parsed command to a privileged action was identified in the inspected caller: the parsed result becomes an evaluation record, not a tool dispatch.

Trust Boundaries and Controls

• observed — The helper accepts tool names from command text and performs parsing, not authorization. The inspected LogParser use does not treat its result as permission to execute a tool.

🚥 Pre-merge checks | ✅ 1

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description includes all required sections and clearly explains the problem, mechanical changes, testing, release status, documentation status, and agent context. The agent context does not includ…

✨ Finishing Touches 💡 1

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

🛠️ Fix failing CI checks 💡

• Commit to this branch
• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}

[trunk-io]

   

Failed Test	Failure Summary	Logs
test_a_batch_that_is_not_final_publishes_nothing	The delivery process failed because the destination warehouse 'a' has no configured integration to connect with.	Logs ↗︎

View Full Report ↗︎ ⋅ Docs

[trunk-io]

Stacked PR 106430 was cancelled: the stack changed after it was queued.

[trunk-io]

This pull request was merged into master as part of stacked PR 106735.

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-09-25 18:17 UTC	Run
prod-us	✅ Deployed	2026-09-25 18:30 UTC	Run
prod-eu	✅ Deployed	2026-09-25 18:32 UTC	Run