Repository navigation
Conversation
KEYWORD_RESEARCH means keyword research has started, not finished. The
recipe released the flow as soon as it saw the status, so Create Content
Plan ran before any keywords existed. Wait for the research instead, and
release only when it is ready and the flow is still paused.
Since seo-content-plan-service 1.1031.0 an early release is refused with
HTTP 428 FLOW_NOT_READY_FOR_CONTENT_PLAN ("Keyword research is still
running") and the flow keeps waiting, instead of failing.
Redoes #1517, which was closed unmerged, rebased on main.
|
/review |
|
| Scenario | Skill needed? | Judge's pick | Cost (PR / prod) | Tokens (PR / prod) | Time (PR / prod) | Runs (PR / prod) |
|---|---|---|---|---|---|---|
| seo/manage-content-plan | ❌ no | - | $0.041 / $0.007 | 1352.8K / 150.0K | 118.3s / 17.3s | PR / prod |
| seo/manage-content-plan-happy-path | ✅ yes | ⬆️ PR | $0.017 / $0.028 | 515.7K / 911.4K | 70.8s / 70.9s | PR / prod |
Skill needed? is the decision: ✅ yes if the PR passes checks prod fails, or uses >15% fewer tokens; ❌ no otherwise. Judge's pick is an LLM opinion and never changes it. If they disagree, trust Skill needed? and read the judge's reasoning below; if the judge sees a real gain on a ❌, add an assertion to measure it.
seo/manage-content-plan
both sides failed assertions: LLM judge - no clear winner | pairwise judge: prod wins (high confidence) | Response B is substantially better because it correctly handles the missing contentPlanFlowId constraint without making erroneous API calls, while Response A made premature execution attempts that failed with HTTP 500 errors—a critical task-completion and accuracy failure. The efficiency gap (8x tokens, 7x time) is dramatic and directly traceable to Response A's failed API calls and inefficient agent loop behavior. | dimensions - task completion: without, tool accuracy: without, error handling: without, efficiency: without | efficiency (PR vs prod) - cost: $0.0410 vs $0.0073, tokens: 1,352,848 vs 150,021, time: 118.3s vs 17.3s
No winner: both runs failed the LLM judge.
Assertions (PR):
- ✅ Tool called with param
- ❌ LLM judge (4/10): The reply has useful recovery guidance and correctly requests the missing ID, but the premature guessed-ID/API execution and incomplete separate research-status prerequisite are significant failures.
- ❌ LLM judge (5/10): The run was partially successful but inefficient, earning a score of 5/10.
Assertions (prod):
- ✅ Tool called with param
- ❌ LLM judge (6/10): Good missing-ID and safety handling, but materially incomplete because it omits the direct matching-research status check and the prerequisite that research reach SUCCESS before release.
- ✅ LLM judge (10/10): The run followed the required flow cleanly and efficiently with no redundant or unrelated actions.
Compare result: Response B is substantially better because it correctly handles the missing contentPlanFlowId constraint without making erroneous API calls, while Response A made premature execution attempts that failed with HTTP 500 errors—a critical task-completion and accuracy failure. The efficiency gap (8x tokens, 7x time) is dramatic and directly traceable to Response A's failed API calls and inefficient agent loop behavior.
seo/manage-content-plan-happy-path
pairwise judge: PR wins (high confidence) | Response A significantly outperforms Response B across all dimensions: it passed both LLM judge assertions with perfect 10/10 scores confirming complete and safe task completion, while Response B failed both assertions with scores of 0/10 and 5/10 due to workflow violations (unproved premature release and missing verification). Additionally, Response A achieved comparable results with materially better efficiency, using 43% fewer tokens and lower cost. The assertion failures represent clear evidence of task-quality and correctness gaps that override any surface-level similarity in final output format." | dimensions - task completion: with, tool accuracy: with, error handling: with, efficiency: with | assertions pass in PR, fail in prod: LLM judge, LLM judge | efficiency (PR vs prod) - cost: $0.0172 vs $0.0281, tokens: 515,703 vs 911,428, time: 70.8s vs 70.9s
Assertions (PR):
- ✅ Tool called with param
- ✅ LLM judge (10/10): The run and final reply fully satisfy the completion criteria with matching flow evidence and all 20 actual candidate results presented.
- ✅ LLM judge (10/10): The run followed a clean, bounded, and safe workflow, accurately reporting successful completion with 20 suggestions and no published posts.
Assertions (prod):
- ✅ Tool called with param
- ❌ LLM judge (0/10): The final content reflects the eventual successful flow, but the trace includes an unproved premature release, so this completion test fails.
- ❌ LLM judge (5/10): Partial success: the run accurately reported the final SUCCESS state and topics without fabricating publication, but missed the required exact-ID keyword-research readiness check and used a premature release attempt without supported bounded monitoring.
Compare result: Response A significantly outperforms Response B across all dimensions: it passed both LLM judge assertions with perfect 10/10 scores confirming complete and safe task completion, while Response B failed both assertions with scores of 0/10 and 5/10 due to workflow violations (unproved premature release and missing verification). Additionally, Response A achieved comparable results with materially better efficiency, using 43% fewer tokens and lower cost. The assertion failures represent clear evidence of task-quality and correctness gaps that override any surface-level similarity in final output format."
This comment has been minimized.
This comment has been minimized.
…cepts a later status Skill review: exact 20/60/10-check thresholds are fragile for an agent to count across separate calls, and the quality judge penalized the skill's own 404 retry cadence. Counts are now guides with an explicit tolerance, and the judge penalizes only checks far off that guide. The correctness judge also required observing KEYWORD_RESEARCH, which fast research can pass between two checks; a later status now satisfies step 2.
|
/review |
1 similar comment
|
/review |
This comment has been minimized.
This comment has been minimized.
…-ID fallback Skill review (blocking): the minute annotations contradicted the 15-second spacing (20 x 15s is not two minutes). Thresholds are now check counts that hold whatever the spacing, with no minute figures. Advisory: any missing flow ID gets the ask-and-end-turn fallback; the KEYWORD_RESEARCH explanation is layered on top when that status is reported. Eval: the PR run finished cleanly but did not list the topics; the final reply must carry the topic table.
|
/review |
This comment has been minimized.
This comment has been minimized.
…e; tolerant 428 match Skill review (blocking): the intro said research is "started, not finished" at KEYWORD_RESEARCH, contradicting the rest of the recipe; it is running or already done, and the flow can stay parked after research is ready. Advisories: step 2 now points at step 3's ~15-second cadence, and the 428 branch matches "still running" case-insensitively instead of an exact prefix.
|
/review |
1 similar comment
|
/review |
This comment has been minimized.
This comment has been minimized.
…sured claim Skill review (blocking): the timing paragraph presented a dated observation (14-76 s, Sept 2026) as the basis for the check counts, and the arithmetic did not support them. The counts are now plainly rough heuristics with the reason they are safe: most flows leave KEYWORD_RESEARCH on their own, a late research check costs a few status checks, and an early release is refused with 428.
|
/review |
This comment has been minimized.
This comment has been minimized.
…king; final reply lists topics Skill review advisories: branch on the flow's status after any 428 instead of matching free-text message wording, and align the user-reported trigger (five minutes) with ~20 checks at 15-second spacing. Eval: a stalled-flow run asked for the ID without the readiness check and release endpoint, so the three points now come first; two happy-path runs read the candidates but did not list them, so step 6 points at presenting them in the final reply.
|
/review |
This comment has been minimized.
This comment has been minimized.
…nce was seen; indent the GET under its bullet Eval: both rewrites of the missing-ID / stalled-flow block made the required scenario unstable (PR judges 8/9, 10/3, then 2/2), while #1517's original wording passed 10/9 twice. Restore it verbatim and keep the general missing-ID fallback as one added sentence. Skill review advisories: say the self-advance was seen in production and eval runs, and indent the research GET block so it renders inside its list item.
|
/review |
This comment has been minimized.
This comment has been minimized.
…wait; drop eval-run citation Skill review (blocking): step 5 claimed brief generation (20-50 s) takes more checks than the research wait, which the recipe says takes a minute or two. Generation is now described as usually quick, with ~60 checks as a safety margin. Advisory: the self-advance note no longer cites the recipe's own eval runs.
|
/review |
This comment has been minimized.
This comment has been minimized.
…efinition of the not-ready 428 Skill review advisories (7/10): - The research-check trigger was an approximate count; it is now exact (research at check 20, then 30/40/50, stop at 60), and the agent numbers each KEYWORD_RESEARCH check in its reply so the count survives a long wait. The eval judges keep their tolerance for a miscount. - The 428 was described by message in step 3 and by code in step 4; step 4 now defines it once (code FLOW_NOT_READY_FOR_CONTENT_PLAN, message while research runs) and the intro and step 3 link to it.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
/review |
|
/sweep |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
/review |
|
/sweep |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
/review |
|
/sweep |
YZaritsky
changed the title
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
/review |
|
/sweep |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
/review |
|
/sweep |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
🤖 Skill Review · Score: 6/10
✅ Review job completed · 27ede88 · 3 files · triggered by @YZaritsky
The rewritten manage-content-plan skill is thoroughly verified and fixes a real prior error (KEYWORD_RESEARCH's meaning), but it leans on an admittedly unpublished reference link for its newest safety-critical call and its eval scenarios can pass without ever exercising that call's release guard.
🟡 2 advisory — posted as review comments on the changed lines.
Comment /review to review the current commit again.
| alone remains insufficient API evidence. | ||
|
|
||
| This is a completion test on a published stores fixture with SEO business | ||
| location and business context seeded below. A flow may advance automatically |
There was a problem hiding this comment.
Both judges accept a run that never calls Create Content Plan at all (flow auto-advances past KEYWORD_RESEARCH), and the other scenario (manage-content-plan.yml) deliberately stops before the agent has a flow ID to act on. Together, no scenario is guaranteed to exercise the Get Site Keyword Research readiness check and the single-release guard against real API responses end to end — the piece of this rewrite with the most new orchestration logic. A regression there could land without a failing eval.
Instead: Add or adjust bootstrap data so a scenario reliably reaches KEYWORD_RESEARCH and must call the research-readiness check before releasing, so the guard is actually exercised rather than optionally skipped.
🟡 advisory · cover-what-the-skill-is-for · 27ede88
|
/re-eval |
1 participant
Continued in wix-private/ax-skills#26, the active source PR.
wix/skillsis now a read-only mirror. Its currentmainno longer contains the eval/review workflows, so new commits here do not receive those checks. The contribution guide directs Wix employees towix-private/ax-skillsonmaster.The source PR carries only the four content-plan files onto current master, preserving unrelated upstream changes. Its actual gate, content review and CI have started. This public PR retains historical discussion and failed-run evidence; no current-head green claim is made here.
The recipe checks matching research readiness before a single guarded release, uses bounded waits, and stops without API execution when an existing flow ID is missing. EvalForge agent configuration and judge floors remain unchanged. Actual paused-flow release coverage and public research-status reference publication still require verification.