feat(automations): graphs and mappings (2/6) by yehonatanpeleg · Pull Request #1648 · wix/skills

yehonatanpeleg · 2026-10-07T10:37:13Z

Part 2/4 of the Automations skill publication. Graph model, trigger configuration, app actions and bracket mappings.

Each PR adds exactly four reference Markdown files relative to its immediate predecessor, satisfying the current four-file gate. Documentation YAML, index entries and applicable evals are included. The stack progressively exposes 4/8/12/16 guides and 3/5/21/29 scenarios.

Canonical source: https://github.com/wix-private/crm-automations-client/pull/14563 (e93da2b345). The exporter records the publication stage and preserves it during later syncs. Earlier stages defer entire scenarios until their required guides exist; outcome, quality and persistence assertions are unchanged. Future article links use an explicit unpublished-guide notice and the official API reference until the guide arrives.

Validation: 16 exporter tests passed; this stage has 5 scenarios validated against the repository schema, all registered articles covered, matching frontmatter/index, valid internal article links, and a successful live docs-category check. These checks are structural, not live agent evaluation results.

Merge order: part 1 → part 2 → part 3 → part 4 (original PR #1621). All four parts are ready for review and linked with GitHub’s native stack feature; each retains its predecessor as its base. Merge bottom-up. GitHub’s native stack feature can cascade ancestry/base updates; verify each layer still adds only four new reference Markdown files and inspect actual current-head evaluations before merging. No automatic merge.

The full stack preserves all original 27 scenarios and adds read-only catalog discovery and inactive-status cases. Existing Item Selection external-binding review findings remain unresolved; splitting does not establish API access or live success.

Stack: 1: discovery/lifecycle → 2: graphs/mappings → 3: conditions/timing → 4: providers/email.

github-actions · 2026-10-07T12:27:33Z

⚠️ EvalForge YAML Gate: Eval Comparison

Verdict: required | Tag: draft:wix/skills#1648

Scenario	Skill needed?	Judge's pick	Cost (PR / prod)	Tokens (PR / prod)	Time (PR / prod)	Runs (PR / prod)
automations/create-and-activate-request	❌ no	⬇️ prod	$0.033 / $0.036	962.6K / 998.6K	115.6s / 134.9s	PR / prod
automations/create-and-activate-request	✅ yes	⬆️ PR	$0.029 / $0.027	776.3K / 730.0K	86.7s / 76.9s	PR / prod
automations/create-conditional-title-mapping	✅ yes	⬆️ PR	$0.028 / $0.073	777.7K / 2224.6K	80.1s / 195.9s	PR / prod
automations/create-filtered-label-task-chain	✅ yes	⬆️ PR	$0.026 / $0.062	672.6K / 1987.1K	79.5s / 154.8s	PR / prod
automations/create-numeric-input-mapping	✅ yes	⬆️ PR	$0.039 / $0.076	1061.5K / 2107.0K	99.3s / 176.1s	PR / prod
automations/deactivate-request	✅ yes	⬆️ PR	$0.011 / $0.016	272.9K / 428.5K	35.4s / 51.1s	PR / prod
automations/discover-contact-label-capabilities	❌ no	⬆️ PR	$0.017 / $0.012	311.9K / 319.8K	46.9s / 40.9s	PR / prod
automations/discover-contact-task-capabilities	✅ yes	≈ tie	$0.020 / $0.014	457.9K / 325.1K	73.9s / 50.7s	PR / prod
automations/inspect-active-status	❌ no	⬇️ prod	$0.010 / $0.009	271.9K / 254.4K	25.8s / 22.2s	PR / prod
automations/inspect-ambiguous-name	❌ no	⬇️ prod	$0.010 / $0.009	272.0K / 255.5K	30.6s / 31.1s	PR / prod
automations/inspect-inactive-status	❌ no	⬆️ PR	$0.010 / $0.011	271.6K / 313.7K	29.3s / 30.3s	PR / prod
automations/inspect-missing-automation	❌ no	⬇️ prod	$0.010 / $0.008	271.4K / 254.0K	23.5s / 22.1s	PR / prod
automations/inspect-unknown-name	✅ yes	⬇️ prod	$0.010 / $0.012	271.3K / 372.3K	23.2s / 34.7s	PR / prod

Skill needed? is the decision: ✅ yes if the PR passes checks prod fails, or uses >15% fewer tokens; ❌ no otherwise. Judge's pick is an LLM opinion and never changes it. If they disagree, trust Skill needed? and read the judge's reasoning below; if the judge sees a real gain on a ❌, add an assertion to measure it.

automations/create-and-activate-request

pairwise judge: prod wins (medium confidence) | While Response B provides superior task completion and transparency through more detailed automation configuration information and explicit validation, Response A achieves the same functional outcome with notably better efficiency across all metrics (time, tokens, and cost). The efficiency advantage is material and measurable, but does not outweigh the meaningful completeness gap that makes Response B's output more informative and trustworthy for verifying the automation was set up correctly. | dimensions - task completion: without, tool accuracy: without, efficiency: with | tie: PR tokens within ±15% of prod | efficiency (PR vs prod) - cost: $0.0330 vs $0.0364, tokens: 962,606 vs 998,645, time: 115.6s vs 134.9s

View run (PR)

View run (prod)

Assertions (PR):

Assertions (prod):

Compare result: While Response B provides superior task completion and transparency through more detailed automation configuration information and explicit validation, Response A achieves the same functional outcome with notably better efficiency across all metrics (time, tokens, and cost). The efficiency advantage is material and measurable, but does not outweigh the meaningful completeness gap that makes Response B's output more informative and trustworthy for verifying the automation was set up correctly.

automations/create-and-activate-request

pairwise judge: PR wins (high confidence) | Response A is meaningfully better because it fully satisfied all documented requirements with perfect LLM judge assertions (3/3 passed), while Response B failed critical lifecycle and validation procedures (2/3 failed). The assertion failures are strong evidence of task-quality gaps that outweigh Response B's minor efficiency advantage, indicating Response A correctly completed the full automation creation and activation workflow as required. | dimensions - task completion: with, tool accuracy: with, error handling: with, efficiency: without | assertions pass in PR, fail in prod: LLM judge, LLM judge | efficiency (PR vs prod) - cost: $0.0287 vs $0.0266, tokens: 776,255 vs 729,957, time: 86.7s vs 76.9s

View run (PR)

View run (prod)

Assertions (PR):

✅ Tool called with param
✅ Tool called with param
✅ Tool called with param
✅ Tool called with param
✅ Tool called with param
✅ Tool called with param
✅ LLM judge (10/10): The automation was correctly created, activated, and configured as required without unnecessary or unrelated operations.
✅ LLM judge (10/10): The run fully satisfied the documented criteria and successfully saved and activated the requested automation.
✅ LLM judge (10/10): The automation was correctly saved, enabled, and clearly reported as active with the required details.

Assertions (prod):

✅ Tool called with param
✅ Tool called with param
❌ Tool called with param
❌ Tool called with param
❌ Tool called with param
❌ Tool called with param
❌ LLM judge (0/10): The run does not satisfy the required automation creation, validation, update, and verification sequence.
❌ LLM judge (5/10): The automation was created and activated successfully, but the run only partially satisfied the required lifecycle and catalog-discovery procedure.
✅ LLM judge (10/10): The response fully satisfies the requirements.

Compare result: Response A is meaningfully better because it fully satisfied all documented requirements with perfect LLM judge assertions (3/3 passed), while Response B failed critical lifecycle and validation procedures (2/3 failed). The assertion failures are strong evidence of task-quality gaps that outweigh Response B's minor efficiency advantage, indicating Response A correctly completed the full automation creation and activation workflow as required.

automations/create-conditional-title-mapping

pairwise judge: PR wins (high confidence) | Response A is materially superior across all dimensions. It fully satisfies the task requirements with perfect assertion scores, correct implementation, and efficient execution. Response B fails critical assertions on fallback behavior and overall criteria, required inefficient trial-and-error to arrive at a flawed solution, and consumed 3x more tokens and time. The assertion failures provide strong evidence of task completion gaps beyond surface-level output differences. | dimensions - task completion: with, tool accuracy: with, error handling: with, efficiency: with | assertions pass in PR, fail in prod: LLM judge, LLM judge, LLM judge | efficiency (PR vs prod) - cost: $0.0283 vs $0.0733, tokens: 777,748 vs 2,224,608, time: 80.1s vs 195.9s

View run (PR)

View run (prod)

Assertions (PR):

✅ Tool called with param
✅ Tool called with param
✅ Tool called with param
✅ Tool called with param
✅ Tool called with param
✅ LLM judge (10/10): The automation fully satisfies the requirements with correct action mapping, dynamic first-name handling, fallback behavior, contact ID preservation, and valid expression syntax.
✅ LLM judge (10/10): The run fully satisfied the requirements with no meaningful friction, unsupported claims, or unrequested side effects.
✅ LLM judge (10/10): The automation was correctly validated and created once in an inactive state, and the final answer accurately reported its saved configuration and ID.

Assertions (prod):

✅ Tool called with param
❌ Tool called with param
❌ Tool called with param
❌ Tool called with param
❌ Tool called with param
❌ LLM judge (0/10): The automation does not satisfy the required fallback behavior because its condition does not explicitly handle both missing and empty values.
❌ LLM judge (6/10): The automation was successfully created and activated with the requested behavior, but the implementation required inefficient trial-and-error and repeated repairs before reaching the correct configuration.
❌ LLM judge (0/10): The run fails the required criteria.

Compare result: Response A is materially superior across all dimensions. It fully satisfies the task requirements with perfect assertion scores, correct implementation, and efficient execution. Response B fails critical assertions on fallback behavior and overall criteria, required inefficient trial-and-error to arrive at a flawed solution, and consumed 3x more tokens and time. The assertion failures provide strong evidence of task completion gaps beyond surface-level output differences.

automations/create-filtered-label-task-chain

pairwise judge: PR wins (high confidence) | Response A is materially superior across all dimensions. It passed all three LLM judge assertions (10/10, 9/10, 10/10) and correctly created the automation in INACTIVE status with proper configuration and post-creation verification. Response B failed all three assertions due to critical task violations: creating the automation in ACTIVE instead of INACTIVE status, using incorrect filterExpression form and node namespace numbering, and omitting independent post-read verification. These are not minor stylistic differences—they represent fundamental failures in meeting mandatory requirements. Response A is also substantially more efficient, using 3x fewer tokens and completing in half the time. | dimensions - task completion: with, tool accuracy: with, error handling: with, efficiency: with | assertions pass in PR, fail in prod: LLM judge, LLM judge, LLM judge | efficiency (PR vs prod) - cost: $0.0261 vs $0.0624, tokens: 672,554 vs 1,987,098, time: 79.5s vs 154.8s

View run (PR)

View run (prod)

Assertions (PR):

✅ Tool called with param
✅ Tool called with param
✅ Tool called with param
✅ Tool called with param
✅ Tool called with param
✅ Tool called with param
✅ LLM judge (10/10): The run fully satisfies all mandatory requirements.
✅ LLM judge (9/10): The run followed a direct, bounded, documented path and correctly created the requested automation once, with only minor trace-visibility limitations.
✅ LLM judge (10/10): The run fully satisfies the requirements: the automation was validated and created exactly once in an inactive state with the requested configuration, then read back and correctly reported.

Assertions (prod):

✅ Tool called with param
✅ Tool called with param
❌ Tool called with param
❌ Tool called with param
❌ Tool called with param
❌ Tool called with param
❌ LLM judge (0/10): The run does not satisfy the requirements because the filterExpression has the wrong form and the node namespaces use incorrect numbering.
❌ LLM judge (4/10): The automation was created correctly, but substantial discovery inefficiency, a failed documentation call, disallowed hardcoding, and lack of independent post-create verification justify a score of 4/10.
❌ LLM judge (0/10): The run receives a zero score because it used ACTIVE instead of the required INACTIVE status, omitted a separate post-creation Get Automation read-back, and reported the automation as Active.

Compare result: Response A is materially superior across all dimensions. It passed all three LLM judge assertions (10/10, 9/10, 10/10) and correctly created the automation in INACTIVE status with proper configuration and post-creation verification. Response B failed all three assertions due to critical task violations: creating the automation in ACTIVE instead of INACTIVE status, using incorrect filterExpression form and node namespace numbering, and omitting independent post-read verification. These are not minor stylistic differences—they represent fundamental failures in meeting mandatory requirements. Response A is also substantially more efficient, using 3x fewer tokens and completing in half the time.

automations/create-numeric-input-mapping

pairwise judge: PR wins (high confidence) | Response A is significantly better than Response B. Response A successfully completed all requirements with passing LLM judge assertions across all three evaluation dimensions, while Response B failed critical semantic and functional requirements (disallowed decimal literal usage, wrong automation status, missing verification). The semantic failure in Response B is particularly important as it directly violates task requirements. Additionally, Response B was substantially less efficient, consuming nearly double the tokens and execution time. | dimensions - task completion: with, tool accuracy: with, error handling: with, efficiency: without | assertions pass in PR, fail in prod: LLM judge, LLM judge | efficiency (PR vs prod) - cost: $0.0386 vs $0.0764, tokens: 1,061,524 vs 2,107,024, time: 99.3s vs 176.1s

View run (PR)

View run (prod)

Assertions (PR):

✅ Tool called with param
✅ Tool called with param
✅ Tool called with param
✅ Tool called with param
✅ LLM judge (10/10): The run fully satisfies the requirements and receives a score of 10.
✅ LLM judge (8/10): The run was successful and well-executed overall, with minor inefficiency and one recoverable API argument-shape error.
✅ LLM judge (10/10): The automation was correctly validated and created once in an INACTIVE state, verified by ID, with no prohibited or unrelated actions.

Assertions (prod):

✅ Tool called with param
❌ Tool called with param
❌ Tool called with param
❌ Tool called with param
❌ LLM judge (0/10): The semantic requirement fails because the mapping uses the disallowed decimal literal 0.1 instead of the required divide-by-10 formulation.
✅ LLM judge (7/10): The automation run was successful and largely well executed, but received a 7 due to redundant discovery work and the lack of an explicit post-creation read-back verification.
❌ LLM judge (0/10): The automation was created with the wrong ACTIVE status, lacked the required post-creation Get Automation read-back, and was incorrectly reported as saved and activated.

Compare result: Response A is significantly better than Response B. Response A successfully completed all requirements with passing LLM judge assertions across all three evaluation dimensions, while Response B failed critical semantic and functional requirements (disallowed decimal literal usage, wrong automation status, missing verification). The semantic failure in Response B is particularly important as it directly violates task requirements. Additionally, Response B was substantially less efficient, consuming nearly double the tokens and execution time.

automations/deactivate-request

pairwise judge: PR wins (high confidence) | Response A is meaningfully better across multiple dimensions. While both responses complete the task successfully, Response A demonstrates superior tool accuracy (consistent perfect scores vs. missing verification call), better engineering practices (less overhead, more optimized execution), and substantially better efficiency metrics (2/3 cost reduction, 36% fewer tokens, 31% faster). The judges explicitly identified operational inefficiencies in Response B that are avoided in Response A. | dimensions - tool accuracy: with, error handling: with, efficiency: with | both complete the task, PR tokens 36% lower than prod | efficiency (PR vs prod) - cost: $0.0108 vs $0.0164, tokens: 272,909 vs 428,483, time: 35.4s vs 51.1s

View run (PR)

View run (prod)

Assertions (PR):

✅ Tool called with param
✅ Tool called with param
❌ Tool called with param
✅ LLM judge (10/10): The automation was correctly updated to inactive with the correct ID and site ID, with no unrelated actions.
✅ LLM judge (9/10): Successful run with a direct, verified status update and only minor unnecessary API/documentation friction.
✅ LLM judge (10/10): The response fully satisfies the requirements.

Assertions (prod):

✅ Tool called with param
❌ Tool called with param
❌ Tool called with param
✅ LLM judge (8/10): The automation was correctly set to INACTIVE, but the missing post-update Get Automation call prevents a perfect score.
✅ LLM judge (7/10): Successful and correctly scoped run, but with unnecessary discovery overhead and avoidable PATCH-payload risk.
✅ LLM judge (10/10): The response fully satisfies the requirements: Contact Follow-up was successfully turned off and is reported as INACTIVE with the correct automation ID.

Compare result: Response A is meaningfully better across multiple dimensions. While both responses complete the task successfully, Response A demonstrates superior tool accuracy (consistent perfect scores vs. missing verification call), better engineering practices (less overhead, more optimized execution), and substantially better efficiency metrics (2/3 cost reduction, 36% fewer tokens, 31% faster). The judges explicitly identified operational inefficiencies in Response B that are avoided in Response A.

automations/discover-contact-label-capabilities

pairwise judge: PR wins (medium confidence) | Response A is objectively more complete and accurate, achieving a 10/10 LLM judge score versus Response B's 8/10, with explicit feedback that Response B's output was missing required appId fields for both the trigger and action. However, Response B was materially more efficient in terms of cost and time. The completeness gap is the deciding factor for this task, as appIds are documented requirements that Response A provides and Response B omits. | dimensions - task completion: with, tool accuracy: with, efficiency: without | tie: PR tokens within ±15% of prod | efficiency (PR vs prod) - cost: $0.0169 vs $0.0121, tokens: 311,887 vs 319,819, time: 46.9s vs 40.9s

View run (PR)

View run (prod)

Assertions (PR):

✅ Tool called with param
✅ Tool called with param
✅ LLM judge (10/10): The answer is fully correct, complete, and supported by the trace.
✅ LLM judge (8/10): The run was strong and correctly verified the relevant trigger and action schemas, with only minor inefficiency from broad catalog scans and redundant documentation reads.

Assertions (prod):

✅ Tool called with param
✅ Tool called with param
✅ LLM judge (8/10): The answer is largely correct but misses the required appId for both the trigger and action, preventing a perfect score.
✅ LLM judge (8/10): Strong, correct read-only execution with targeted catalog lookups and schema verification, but marked down for two unnecessary detours and an insufficiently established endpoint provenance.

Compare result: Response A is objectively more complete and accurate, achieving a 10/10 LLM judge score versus Response B's 8/10, with explicit feedback that Response B's output was missing required appId fields for both the trigger and action. However, Response B was materially more efficient in terms of cost and time. The completeness gap is the deciding factor for this task, as appIds are documented requirements that Response A provides and Response B omits.

automations/discover-contact-task-capabilities

pairwise judge: tie | assertions pass in PR, fail in prod: LLM judge | efficiency (PR vs prod) - cost: $0.0199 vs $0.0138, tokens: 457,946 vs 325,084, time: 73.9s vs 50.7s

View run (PR)

View run (prod)

Assertions (PR):

✅ Tool called with param
✅ Tool called with param
✅ LLM judge (10/10): The run fully satisfies the requirements with an accurate, well-supported final answer and no site or automation changes.
✅ LLM judge (10/10): No meaningful detours; the run was complete, efficient, accurate, and mutation-free.

Assertions (prod):

✅ Tool called with param
✅ Tool called with param
✅ LLM judge (10/10): The run fully satisfied the requirements: it correctly identified that no task-creation action is installed while accurately reporting the trigger and available input, with no site mutations.
❌ LLM judge (6/10): The run was mostly correct and safely executed, but it included an unnecessary documentation call and redundant catalog scans.

Compare result: Both responses achieve the core task equally well and pass their critical assertions confirming task completion and accuracy. Response B is moderately more efficient across all metrics (time, tokens, cost), which breaks the tie. However, both responses successfully deliver the required information without site mutations, and Response B's secondary assertion about unnecessary calls doesn't undermine its core accuracy or output quality, making this a medium-confidence decision favoring the leaner production implementation.

automations/inspect-active-status

pairwise judge: prod wins (high confidence) | Response B is materially better because it fully completed the task (perfect 10/10 score) by including the critical detail that ACTIVE status means the automation is eligible for new trigger events, which Response A explicitly omitted (7/10 score). Response B also demonstrates superior efficiency across all metrics. | dimensions - task completion: without, tool accuracy: without, efficiency: without | tie: PR tokens within ±15% of prod | efficiency (PR vs prod) - cost: $0.0099 vs $0.0088, tokens: 271,860 vs 254,406, time: 25.8s vs 22.2s

View run (PR)

View run (prod)

Assertions (PR):

✅ Tool called with param
✅ Tool called with param
✅ LLM judge (7/10): The answer is mostly correct but incomplete because it omits that the automation is eligible for new trigger events.
✅ LLM judge (9/10): The run was highly successful and correctly reported the automation as ACTIVE, with only minor inefficiencies.

Assertions (prod):

✅ Tool called with param
✅ Tool called with param
✅ LLM judge (10/10): The run fully satisfied the requirements without making unsupported claims or performing prohibited actions.
✅ LLM judge (9/10): The run was highly effective and correctly performed the requested read-only automation status inspection, with only minor inefficiency from unnecessary documentation lookups.

Compare result: Response B is materially better because it fully completed the task (perfect 10/10 score) by including the critical detail that ACTIVE status means the automation is eligible for new trigger events, which Response A explicitly omitted (7/10 score). Response B also demonstrates superior efficiency across all metrics.

automations/inspect-ambiguous-name

pairwise judge: prod wins (high confidence) | Response B is meaningfully better because it demonstrates the correct behavior for handling ambiguous resource names by requesting user clarification rather than making assumptions. While Response A reports information accurately, it fails to ask which automation was intended—a critical oversight when multiple resources match. Both pass LLM assertions, but Response B's handling of the ambiguity is substantively more appropriate, and it additionally provides better efficiency metrics. | dimensions - task completion: without, error handling: without, efficiency: without | tie: PR tokens within ±15% of prod | efficiency (PR vs prod) - cost: $0.0102 vs $0.0095, tokens: 272,010 vs 255,486, time: 30.6s vs 31.1s

View run (PR)

View run (prod)

Assertions (PR):

✅ Tool called with param
✅ Tool called with param
✅ LLM judge (7/10): The run was mostly correct but incomplete because it failed to ask the user which automation they meant.
✅ LLM judge (10/10): The run fully satisfied the requirements with a bounded read-only query and accurate reporting of both matching automations and their statuses.

Assertions (prod):

✅ Tool called with param
✅ Tool called with param
✅ LLM judge (10/10): The agent handled the duplicate automations correctly and safely by requesting clarification without performing any action.
✅ LLM judge (10/10): The run fully satisfied the criteria with a score of 10.

Compare result: Response B is meaningfully better because it demonstrates the correct behavior for handling ambiguous resource names by requesting user clarification rather than making assumptions. While Response A reports information accurately, it fails to ask which automation was intended—a critical oversight when multiple resources match. Both pass LLM assertions, but Response B's handling of the ambiguity is substantively more appropriate, and it additionally provides better efficiency metrics.

automations/inspect-inactive-status

pairwise judge: PR wins (medium confidence) | Response A demonstrates more consistent quality across multiple independent judge evaluations, with both judges rating it highly (9-10/10) without reservations. Response B shows inconsistency with one judge giving 10/10 and another giving 8/10, specifically criticizing the clarity of the final output's explanation of status limitations. While efficiency slightly favors Response B, the task-quality gap from more reliable and consistent judge results supports Response A as the better response. | dimensions - task completion: with, tool accuracy: with, error handling: with, efficiency: without | tie: PR tokens within ±15% of prod | efficiency (PR vs prod) - cost: $0.0100 vs $0.0106, tokens: 271,642 vs 313,676, time: 29.3s vs 30.3s

View run (PR)

View run (prod)

Assertions (PR):

✅ Tool called with param
✅ Tool called with param
✅ LLM judge (10/10): The response fully satisfies the requirements with no unsupported claims or actions.
✅ LLM judge (9/10): The run was highly successful, with only a minor deduction for redundant preliminary documentation/README calls.

Assertions (prod):

✅ Tool called with param
✅ Tool called with param
✅ LLM judge (10/10): The run fully satisfied the criteria with no unsupported claims or unintended write, activation, or test actions.
✅ LLM judge (8/10): The run was mostly correct, but the truncated final output failed to clearly explain the limitation of the INACTIVE status.

Compare result: Response A demonstrates more consistent quality across multiple independent judge evaluations, with both judges rating it highly (9-10/10) without reservations. Response B shows inconsistency with one judge giving 10/10 and another giving 8/10, specifically criticizing the clarity of the final output's explanation of status limitations. While efficiency slightly favors Response B, the task-quality gap from more reliable and consistent judge results supports Response A as the better response.

automations/inspect-missing-automation

pairwise judge: prod wins (high confidence) | Response B is meaningfully better across multiple dimensions. The LLM judge awarded it 10/10 for full task completion versus Response A's 7/10 for partial completion. The critical difference is that Response B explicitly asks the user to confirm the site and automation IDs (addressing the main gap identified in Response A's assessment), provides clearer output structure with the IDs displayed, and achieves this with better efficiency metrics. The assertion evidence directly supports Response B as the superior approach. | dimensions - task completion: without, error handling: without, efficiency: without | tie: PR tokens within ±15% of prod | efficiency (PR vs prod) - cost: $0.0096 vs $0.0084, tokens: 271,393 vs 253,968, time: 23.5s vs 22.1s

View run (PR)

View run (prod)

Assertions (PR):

✅ Tool called with param
✅ Tool called with param
✅ LLM judge (7/10): The agent partially fulfilled the task: it safely reported that the automation could not be found, but failed to ask the user to confirm the site and automation ID.
✅ LLM judge (9/10): The run was highly efficient and appropriate, with only minor extra documentation lookups.

Assertions (prod):

✅ Tool called with param
✅ Tool called with param
✅ LLM judge (10/10): The response fully satisfied the requirements and earned a score of 10.
✅ LLM judge (9/10): The run was highly effective and correct, with only minor unnecessary documentation-reading overhead.

Compare result: Response B is meaningfully better across multiple dimensions. The LLM judge awarded it 10/10 for full task completion versus Response A's 7/10 for partial completion. The critical difference is that Response B explicitly asks the user to confirm the site and automation IDs (addressing the main gap identified in Response A's assessment), provides clearer output structure with the IDs displayed, and achieves this with better efficiency metrics. The assertion evidence directly supports Response B as the superior approach.

automations/inspect-unknown-name

pairwise judge: prod wins (medium confidence) | Response B provides superior task completion and error handling by identifying that the site is in Draft status and explaining its implications, which is more valuable diagnostic information than Response A's generic suggestion. However, Response A is notably more efficient. The LLM judge scores are equivalent for both responses (10/10 and 9/10), but Response B's additional contextual insight about the Draft status provides clearer guidance to users. The efficiency advantage of Response A is not dramatic enough to overcome the better diagnostic quality of Response B. | dimensions - task completion: without, error handling: without, efficiency: with | both complete the task, PR tokens 27% lower than prod | efficiency (PR vs prod) - cost: $0.0095 vs $0.0119, tokens: 271,335 vs 372,325, time: 23.2s vs 34.7s

View run (PR)

View run (prod)

Assertions (PR):

✅ Tool called with param
✅ Tool called with param
✅ LLM judge (10/10): The run fully satisfied the requirements without any prohibited actions.
✅ LLM judge (9/10): The run is highly compliant with the required query behavior, with only unnecessary preliminary documentation/README calls.

Assertions (prod):

✅ Tool called with param
✅ Tool called with param
✅ LLM judge (10/10): The agent correctly handled the missing automation without inventing details or making changes, and provided appropriate next steps.
✅ LLM judge (9/10): The run was highly compliant and correct, with only somewhat extensive documentation/schema discovery as a minor drawback.

Compare result: Response B provides superior task completion and error handling by identifying that the site is in Draft status and explaining its implications, which is more valuable diagnostic information than Response A's generic suggestion. However, Response A is notably more efficient. The LLM judge scores are equivalent for both responses (10/10 and 9/10), but Response B's additional contextual insight about the Draft status provides clearer guidance to users. The efficiency advantage of Response A is not dramatic enough to overcome the better diagnostic quality of Response B.

…efusal scenario (2/6) Re-exported from the canonical source after the skill-review advisories: - the entry guide tells the agent not to repeat rollout terms (stage, not yet published) to the user and to explain limits in product terms; - the scope is stated once at the top of the entry guide; - new scenario create-and-activate-request checks that a save-and-turn-on request is saved inactive with Activate explained. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

yehonatanpeleg · 2026-10-11T10:08:01Z

/review

Re-exported from the canonical source: the create hand-off opens with saved-and-inactive and how to turn it on (Activate, edit link) before the details. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…te evidence (2/6) Re-exported from the canonical source: stage 2 says to create a turn-on request INACTIVE and point to Activate; the create scenarios read-back judges check the Create request, a separate Get and the answer, since judges do not see tool response bodies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ow in Inbox (2/6) Re-exported from the canonical source: the status guide ends a multiple-match listing with a direct question, the create hand-off has a concrete first line, and the Inbox scenarios name the Show in Inbox step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Re-exported from the canonical source: the stage-2 scope explains that the guide has no update or status-change procedure and forbids Update even for explicit turn-on requests or repairs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… (2/6) Re-exported from the canonical source (wix-private/crm-automations-client#14563). Addresses review: stage 2 now publishes the full activation guide (turn on/off on request) instead of refusing, the status guide shows its Query/Get/PATCH calls, and the four stage-2 create scenarios use plain user prompts that do not name the action or fields. create-and-activate-request now expects the automation to end up active. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…rder (2/6) Re-exported from wix-private/crm-automations-client#14563: the hand-off for a turned-on automation opens with its ID, and the activation judge requires Validate only before the Update. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…(2/6) Re-exported from wix-private/crm-automations-client#14563 after running the skill-review agent locally: site lock asks only when the site is ambiguous, opaque widgets defined, read-only capability route, full status-call URLs, new deactivate-request scenario, create judges require Validate before Create. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…nistic deactivate seed (2/6) Re-exported from wix-private/crm-automations-client#14563: last local skill-review round (advisories only). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…a separate Validate; ID on the first hand-off line (2/6) Re-exported from wix-private/crm-automations-client#14563 (e87197249e). Fixes the three create scenarios that failed the eval gate on 001245d; local skill-review: blocking findings fixed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

… the values in the title; percentage example (2/6) Re-exported from wix-private/crm-automations-client#14563 (31aca84037). Judges graded truncated trace previews as the final answer; local skill-review: no blocking findings. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…rs[], never invented (2/6) Re-exported from wix-private/crm-automations-client#14563 (2efb1a226f). Local skill-review: no blocking findings. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…r-chosen ids, encoded literals and abbreviated read-backs as expected (2/6) Re-exported from wix-private/crm-automations-client#14563 (d5f557ecb9). Local skill-review: no blocking findings. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…in judges name the catalog filter id (2/6) Re-exported from wix-private/crm-automations-client#14563 (8e746de0ef). Local skill-review: no blocking findings. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

guyshaulsky · 2026-10-13T10:00:04Z

/review

github-actions

🤖 Skill Review · Score: 8/10

_{✅ Review job completed · af1847d · 19 files · triggered by @github-actions[bot]}

A well-structured, internally-tested expansion of the synced Automations skill area — orchestrator and references stay consistent with each other and with the new eval scenarios, which assert exact call shapes, ordering, and user-facing phrasing with real teeth; only a minor advisory on where the 'don't leak stage language' instruction lives.

🟡 1 advisory — posted as review comments on the changed lines.

Comment /review to review the current commit again.

github-actions · 2026-10-13T10:03:37Z

+description: "Configure app-defined automation actions from their input schemas and preserve supported existing action mappings."
+---
+
+> **Stage scope:** this stage covers only what [Build and Manage Wix Automations](https://dev.wix.com/docs/api-reference/business-management/automations/skills/build-and-manage-wix-automations) lists; do not perform anything marked "Later stages only" or any workflow that needs a guide marked "not yet published".


Every sub-reference (actions.md, triggers.md, automation-model.md, bracket-expressions.md, activation-status.md) repeats 'stage'/'not yet published' language, but the instruction not to say these words to the user lives only once, in build-and-manage-automations.md's body text. The index marks these files 'do NOT use directly', which mitigates the risk, but an agent that reaches one of these references on its own (a stale link, a truncated context that drops the orchestrator) has no local reminder and may surface internal rollout vocabulary straight to the user.

Instead: Add a one-line 'never mention stage/not yet published to the user' reminder to each reference file's stage-scope note, not only to the orchestrator.

_{🟡 advisory · orchestration-and-worked-examples · af1847d}

Brings the branch up to date with main; the exported automations skill is unchanged (same source as wix-private/crm-automations-client#14563 at 8e746de0ef). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

github-actions · 2026-10-13T10:06:06Z

⏳ Awaiting skill review — commit 2f410ef

The review runs automatically when a PR is opened, and on request after that.

Comment /review to review this commit.

…t labels to the user (2/6) Re-exported from wix-private/crm-automations-client#14563 (0e8819b61b): answers the skill-review advisory on actions.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Brings the branch up to date with main; the exported automations skill is unchanged (same source as wix-private/crm-automations-client#14563 at 0e8819b61b). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

yehonatanpeleg · 2026-10-13T11:48:39Z

#skipreview

Brings the branch up to date with main; the exported automations skill is unchanged (same source as wix-private/crm-automations-client#14563 at 0e8819b61b). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…ge 3 (3/6) Re-exported from wix-private/crm-automations-client#14563 (0e8819b61b) onto main, which now contains stage 2 (#1648). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

yehonatanpeleg added 2 commits October 7, 2026 13:30

feat(automations): publish discovery and lifecycle guides (1/4)

add5795

feat(automations): publish graph and mapping guides (2/4)

a747e23

This was referenced Oct 7, 2026

feat(automations): discovery and lifecycle guides (1/6) #1647

Merged

feat(automations): conditions timing and validation (3/6) #1649

Closed

feat(automations): providers and email actions (4/6) #1621

Closed

yehonatanpeleg added 2 commits October 7, 2026 14:00

fix(automations): scope first stage to inspection and verified discovery

34c09a3

fix(automations): propagate validated discovery publication fixes

ad7316c

yehonatanpeleg marked this pull request as ready for review October 7, 2026 12:05

yehonatanpeleg requested a review from a team as a code owner October 7, 2026 12:05

yehonatanpeleg added this pull request to stack #1658 October 7, 2026 12:06

yehonatanpeleg added 2 commits October 7, 2026 15:25

fix(automations): clarify supplied-log diagnosis and revision lookup

193c651

fix(automations): propagate diagnosis and revision corrections

4e9f8b4