feat(flows): workflow-builder agent-friendliness plumbing — schema-aware sandbox, unified draft handles, reference gates#4881
Conversation
…ent nodes The vendored agent node only routes to an AgentRunner when the node carries an agent_ref; builder-generated agent nodes carry none, so in the dry-run sandbox they hit the plain MockLlm whose echo can never satisfy a declared output_parser.schema. The sub-port then failed with 'output_parser: value failed schema validation after auto-fix' on correctly-built graphs (SchemaAwareMockAgentRunner from tinyhumansai#4586 covered only the agent_ref path). Add SchemaAwareMockLlm mirroring the runner's placeholder synthesis and wire it into DryRunWorkflowTool and the propose/save gate sandbox (validate_required_arg_resolvability), which was rejecting the same correct graphs at propose_workflow time.
…tence state on builder tools
WS2 of the workflow_builder audit fix. The builder tools had an asymmetric,
broken state model: edit_workflow{flow_id} applied ops to the saved graph but
persisted nothing and returned no handle, so the edit was unreachable and read
as "written onto the flow"; validate/dry_run/save each accepted a different
subset of handles. An agent wasted 6 of 9 tool calls believing an edit was
saved when it wasn't.
Unified acceptance matrix (all with draft_id > flow_id precedence):
tool before after
---- ------ -----
edit_workflow draft|flow|graph draft|flow|graph (flow seeds a NEW draft)
validate_workflow flow|graph draft|flow|graph
dry_run_workflow draft|graph draft|flow|graph
save_workflow flow_id + inline graph flow_id (target) + draft|graph source
Core fix: edit_workflow{flow_id} now seeds a NEW draft from the flow's graph
(via flows_draft_create, linked to the flow), applies the ops, writes them back
to that draft, and returns draft_id + flow_id + persisted:false + a one-line
`next` hint pointing the agent at the draft to iterate on and save_workflow
{flow_id, draft_id} to persist. The saved flow is never touched.
Explicit persistence state: build_builder_proposal now stamps persisted:false
unconditionally and echoes optional draft_id/flow_id context; propose_workflow's
inline payload and save_workflow's output (persisted:true) match, so no proposal
can read as a save confirmation.
Tests: edit-by-flow seeds a retrievable draft marked unpersisted with the flow
untouched; dry_run/validate by the new handles; save by draft_id persists the
draft graph onto the flow; propose/revise proposals carry persisted:false.
72 builder_tools + tools_tests pass.
…n uncurated actions
Closes two transcript-audit validation gaps that let broken graphs pass every
author-time gate.
1. connection_ref was never validated. The user's connections were
twitter → composio:twitter:ca_JX6QU88UfSk4, gmail → composio:gmail:ca_vX_WA8FsqNmE,
tiktok → composio:tiktok:ca_LPCp3WQpaDma. The agent wired
composio:twitter:ca_LPCp3WQpaDma and composio:gmail:ca_LPCp3WQpaDma (the TIKTOK
id) onto the Twitter and Gmail tool_call nodes; dry_run/validate/propose all
returned ok and it failed on the first real run. New hard gate
`validate_connection_refs` (run right after the binding gate in
`run_builder_gates`): for every Composio tool_call node with a connection_ref
it parses composio:<toolkit>:<id>, rejects malformed refs, enforces the
toolkit segment matches the slug's toolkit, and — when the live connection
list is reachable — that the id names a real connected account OF THAT
toolkit, naming the correct ref verbatim ("did you mean ...") or pointing at
composio_connect when the toolkit has no connection. A missing connection_ref
stays allowed (prompts at first run). Fail-open on connection-list I/O error:
the id-existence check is skipped (tracing::debug) while the no-I/O
toolkit-mismatch check still runs.
2. The curated-action blocker fired too late. get_tool_contract returned
is_curated:false silently; the agent built and wired the node and only ~15
tool calls later did validate_workflow reject it. get_tool_contract now
attaches a top-level `runtime_gate` string (first in the payload) for a
real-but-uncurated action of a toolkit that ships a curated catalog, and
search_tool_catalog flags such rows with `runtime_gated: true` (only when
true). Both reuse a new shared `toolkit_has_curated_catalog` helper so the
author-time gate and the early warnings cannot drift.
… diagnostics, ordering hint
Pre-existing breakage on main: tinyhumansai#4874 added the list_all_runs controller but the schema-coverage tests (count + expected name lists) were not updated, so flows::schemas::tests fails on a clean checkout. Fix the assertions to include it.
…widened agent.toml Pre-existing breakage on main: tinyhumansai#4876 widened workflow_builder's tool belt from 18 to 29 tools (editing, drafts, node-kind introspection, run debugging, create/duplicate) but the loader scope test still asserted the old 18-tool list, so it fails on a clean checkout.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 17 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (11)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
openhuman/src/openhuman/flows/builder_tools.rs
Lines 1141 to 1144 in dc8fd6c
Fail if agent-created flows cannot be disabled
For manual/no-auto-trigger graphs flows_create returns an enabled flow, and this follow-up disable is what enforces the tool's "born disabled" safety contract. If flows_set_enabled(..., false) fails after creation, the code only logs a warning and still returns "enabled": false, leaving the saved flow enabled while telling the agent/user it is disabled; this should surface an error or otherwise avoid reporting success in that failure mode.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| match ops::flows_draft_get(&self.config, id) { | ||
| Ok(outcome) => outcome.value.graph, |
There was a problem hiding this comment.
When save_workflow is called with a draft_id, this branch discards the loaded FlowDraft metadata and keeps only its graph, so the later flows_update(&flow_id, ...) writes that draft onto whichever flow_id the agent supplied. If a draft seeded from flow A is later saved with a stale or copied flow B id, B is silently overwritten with A's draft instead of rejecting the mismatch or using the draft's own flow_id.
Useful? React with 👍 / 👎.
| match ops::flows_draft_get(&self.config, id) { | ||
| Ok(outcome) => outcome.value.graph, |
There was a problem hiding this comment.
For the documented edit_workflow → save_workflow { draft_id, flow_id } path, edit_workflow can update the draft's name, but this save path only extracts outcome.value.graph and leaves name as the separate optional argument. If the agent renamed the draft during editing and then saves by draft_id without repeating name, the graph is persisted while the saved flow keeps its old name, unlike flows_draft_promote, which promotes draft.name.
Useful? React with 👍 / 👎.
1 participant
Summary
workflow_buildersessions showed ~half of all tool calls wasted on tooling friction, and neither build session ever achieved a clean dry-run. This PR fixes the seven root causes found.output_parser.schemafor plain agent nodes (the fix(flows): builder produces resolvable graphs (dry-run catches null args + correct envelope binding) #4586 fix only coveredagent_refnodes), eliminating the falseoutput_parser: value failed schema validationfailures.draft_id | flow_id | graphon edit/validate/dry_run (+draft_idsource on save) — andedit_workflow { flow_id }seeds a real draft and returnsdraft_id+persisted: falseinstead of a dead-end proposal that reads like a save.connection_refs whose toolkit doesn't match the slug or whose id isn't a real connection of that toolkit (an agent shipped a TikTok connection id on both Twitter and Gmail nodes with every gate green).runtime_gatewarning on uncurated actions atget_tool_contract/search time, per-keyword fallback ranking for multi-wordsearch_tool_catalogqueries, per-op-indexedit_workflowparse errors +node_idalias + ordering hint, and honest "unverifiable in sandbox" diagnostics for bindings to Composio tool_call outputs.workflow_builder/prompt.mdsynced with the new surface; two pre-existing test breakages onmainfixed (from feat(flows): Workflows page overhaul — canvas builder, list redesign, Runs/Discoveries sub-pages, privacy cleanup #4874 and feat(flows): agent-friendliness — editing, drafts, safety rails, wider tools #4876).Problem
Live session transcripts (2026-07-15, staging) show the workflow_builder agent:
agent_refroute to the plainMockLlmin the sandbox (vendor/tinyflows/src/nodes/integration/agent.rsrun_turn), whose echo can never satisfy a declared schema — theSchemaAwareMockAgentRunnerfrom fix(flows): builder produces resolvable graphs (dry-run catches null args + correct envelope binding) #4586 never fires for them. One session "fixed" it by inserting fallback-literal transform nodes that degrade the real runtime graph.edit_workflow { flow_id }persisted its edit (the result is stamped with the real flow id/name, carries no handle and no persistence flag), thenvalidate_workflow { flow_id }validated the old saved graph and returnedok: true, reinforcing the wrong belief — 6 of 9 calls wasted in that session.connection_refat all.TWITTER_USER_LOOKUP_MEfor ~15 calls before the curated-actions gate finally rejected it at validate time.edit_workflowop-shape guesses (node_idvsid; add-before-remove ordering).search_tool_catalogqueries returncount: 0with no fallback (GMAIL_FETCH_EMAILSexists).Solution
One commit per workstream:
fix(flows): sandbox mock LLM honors output_parser.schema for plain agent nodes— newSchemaAwareMockLlm(mirrors the runner's placeholder synthesis; vendored echo shape when no schema), wired intoDryRunWorkflowToolandvalidate_required_arg_resolvability(which is whypropose_workflowwas also rejecting correct graphs).feat(flows): unified draft_id|flow_id|graph handles + explicit persistence state— acceptance matrix unified withdraft > flow > graphprecedence;edit_workflow { flow_id }seeds a new draft via the same op the canvas uses, never mutates the flow, and returnsdraft_id/flow_id/persisted: false/anexthint; proposals always carrypersisted: false;save_workflowtakesdraft_idas source.feat(flows): connection_ref hard gate + early runtime-gate warnings—validate_connection_refsinrun_builder_gates(toolkit-mismatch check is I/O-free; existence check is scoped to the matching toolkit and fail-open when the connection list is unavailable);get_tool_contractputs a top-levelruntime_gatewarning on uncurated actions of curated toolkits;search_tool_catalogrows carryruntime_gated: true.feat(flows): agent-friendly edit_workflow op errors— serdenode_id/new_node_idaliases in the vendoredGraphOp, element-by-element parsing with op index + expected shape in errors, ordering hint on add-node-exists. Bumps thevendor/tinyflowssubmodule (branchfeat/edit-workflow-op-aliases, pushed).feat(flows): per-token fallback ranking for search_tool_catalog— 0-hit multi-token queries fall back to per-keyword OR scoring (top 10, curated tiebreak) with an explanatorynote; single-token behavior unchanged.feat(flows): honest sandbox diagnostics for Composio-upstream bindings— null required args whose expression targets a Composiotool_callnode are flaggedunverifiable: true+upstream_tool_callwith pointers toget_tool_contractoutput_fields /get_tool_output_sample; the propose/save gate no longer blocks on this class (dry-run remains the reporting surface).docs(flows): sync workflow_builder prompt— sandbox table, handle matrix, per-op shapes, new gates (+59 lines, every claim verified against code).test(...)commits fix pre-existing breakage onmain: feat(flows): Workflows page overhaul — canvas builder, list redesign, Runs/Discoveries sub-pages, privacy cleanup #4874 added thelist_all_runscontroller without updating the schema-coverage assertions; feat(flows): agent-friendliness — editing, drafts, safety rails, wider tools #4876 widened workflow_builder's tool belt to 29 without updating the loader scope test. Both fail on a cleanmaincheckout.Tradeoffs: the dry run still returns
ok: falseon unverifiable Composio-upstream bindings (silently passing could hide typos) — only the propose/save hard gate downgrades them; the connection-existence check fails open when the connections backend is unreachable so offline/test runs don't false-positive.Submission Checklist
flows:: + tinyflows::caps + agent_registry loader: 573 passed; vendored crate: 340 passed)N/A: agent-tooling behavior change, no feature rows added/removed## Related—N/A: no matrix rows affectedN/A: no release-cut surface touchedCloses #NNN—N/A: no tracking issue; driven by live transcript auditImpact
vendor/tinyflowssubmodule pointer advances toe5327de(pushed totinyhumansai/tinyflowsbranchfeat/edit-workflow-op-aliases).edit_workflow { flow_id }now creates a draft (visible to the canvas) instead of returning a detached proposal.Related
flow_discoveryagent sometimes ends without calling its terminalsuggest_workflowstool (seen in the same transcript audit); semantic catalog ranking beyond the per-token fallback.AI Authored PR Metadata (required for Codex/Linear PRs)
Linear Issue
Commit & Branch
Validation Run
pnpm --filter openhuman-app format:check— N/A (no app changes)pnpm typecheck— N/A (no TS changes)cargo test --lib -- flows:: tinyflows::caps agent_registry::agents::loader→ 573 passed / 0 failed;cargo testin vendor/tinyflows → 340 passedcargo fmt+GGML_NATIVE=OFF cargo checkclean (pre-existing warnings only)Validation Blocked
command:N/Aerror:N/Aimpact:N/ABehavior Changes
Parity Contract
connection_refstill allowed (prompts at first run); UI/RPC canvas save path stays permissive.save_workflowinline-graph path unchanged.Duplicate / Superseded PR Handling