docs(industrial-embodied-ai): show that an individually safe motion is not a trusted one (#27)

Author: carloshvp

industrial-embodied-ai/README.md

@@ -19,21 +19,36 @@ to produce durable evidence:
 
 1. **Allowed and completed:** cMCP authorizes the declared workflow, then the
    independent controller accepts and completes the simulated motion.
-2. **Policy denied:** cMCP denies an undeclared workflow before the controller
-   receives the request.
+2. **Scope denied:** the agent requests a motion whose physical parameters sit
+   inside the safety envelope (an approved zone, an in-limit speed), but under
+   an undeclared workflow. cMCP denies it on scope before the controller is
+   consulted. Physical safety was never the question: the trust layer withholds
+   the action because it falls outside the agent's declared purpose.
 3. **Safety rejected:** cMCP authorizes the declared workflow, but the
    controller rejects motion after its current state reports a person in the
    safeguarded area.
 4. **Closed-session evidence:** cMCP signs a TRACE Trust Record and audit
    bundle that can be verified from the saved files without a running agent,
    runtime or controller.
 
-The third path is the central boundary:
+Two boundaries sit at the center of this example, and they point in opposite
+directions:
 
 > A cMCP `allow` decision means the software request is authorized. It does
 > not mean that a physical action is safe, accepted by the controller, or
 > completed by a machine.
 
+> A controller `accept` decision means a motion is inside the safety envelope.
+> It does not mean the action was authorized, in declared scope, or issued by
+> the agent that was reviewed. The safety controller has no concept of
+> workflow, declared purpose, or agent identity, so it cannot answer those
+> questions. The trust layer is what answers them.
+
+The third path shows the first boundary: an authorized request the controller
+still refuses. The second path shows the second boundary: an in-envelope motion
+the trust layer refuses on scope. Neither layer subsumes the other, and an
+individually safe motion is not, by itself, a trusted one.
+
 ## Architecture
 
 ```text
@@ -75,13 +90,35 @@ rather than a native runtime binding.
 |---|---|
 | Declared agent configuration | A signed Agent Manifest declares the development agent identity and hashes for its prompt, policy and tools |
 | Governed tool access | cMCP intercepts each MCP request and evaluates the active Cedar policy before forwarding |
+| Scope over safety | cMCP denies an in-envelope motion that falls outside the declared workflow, a check the safety controller does not perform |
 | Physical authority | The independent controller rechecks current state and remains authoritative for simulated execution |
 | Durable session evidence | TRACE and the signed audit bundle bind the cMCP session, policy, catalog and tool-call transcript |
 
 The example composes these boundaries without claiming that the developer
 preview already forms one end-to-end identity and outcome proof. The precise
 gaps are listed under [Evidence boundaries](#evidence-boundaries).
 
+## Why individually safe motions still need trust
+
+A safety controller answers one question per motion: is this movement, right
+now, inside the envelope. It cannot answer whether a long run of individually
+safe motions adds up to something the operator never authorized: an agent that
+quietly skips an inspection step, drifts from its declared task, or runs a
+configuration that no longer matches the one that was reviewed. Every motion
+passes. The harm is in the pattern, not in any single move, and no safety
+controller is built to see it.
+
+That is why the durable evidence matters as much as the live decision. The
+signed TRACE record and audit bundle let a later reader, an auditor or an
+insurer who has no reason to trust the operator's database, reconstruct which
+agent configuration acted, under which policy, and against which declared scope,
+across the whole session. The trust layer is not a second safety check. It
+answers a different question, on a different clock, that physical safety alone
+cannot. The controller-side of this boundary is pinned by
+`test_safe_motion_is_not_proof_of_authorization` in `tests/test_controller.py`:
+a motion the controller accepts as safe carries no claim about scope, purpose
+or agent identity.
+
 ## Run it
 
 Prerequisites:
@@ -216,6 +253,7 @@ The validator checks:
 |---|---|---|
 | Agent Manifest | The signed agent identity declaration and hashes of the approved prompt, policy and tools | That cMCP loaded the manifest or bound its agent identity to the runtime session |
 | cMCP decision | The active policy authorized or denied a cataloged tool request | That an authorized physical request was safe |
+| Controller accept | The motion was inside the safety envelope for this run | That the action was authorized, in declared scope, or issued by the reviewed agent |
 | TRACE Trust Record | cMCP session identity, runtime, policy hash, catalog hash and tool-call transcript integrity | The Agent Manifest identity, controller acceptance, physical completion or functional-safety compliance |
 | Saved TRACE and audit files | The closed session can be checked after the processes stop | Continuity of agent memory, reputation or logical identity across a restart or replacement |
 | Client-observed controller response | The mock controller's decision returned to the agent during this run | A signed, hardware-backed or independently retained execution record |

industrial-embodied-ai/tests/test_controller.py

@@ -102,6 +102,31 @@ def test_target_must_be_in_approved_zone(self) -> None:
                 self.request(snapshot["state_token"], target="loading-dock")
             )
 
+    def test_safe_motion_is_not_proof_of_authorization(self) -> None:
+        """An individually safe motion is not, by itself, a trusted one.
+
+        The controller accepts a motion purely on its physical safety
+        envelope: a fresh state token, an approved zone, an in-limit speed and
+        no human present. It has no field for workflow, declared purpose or
+        agent identity, so it cannot tell an in-scope motion from an
+        out-of-scope one. Catching the out-of-scope case is the trust layer's
+        job (the workflow-scoped Cedar policy in policy/allow.cedar), not the
+        safety controller's. This test pins the controller side of that
+        boundary: a safe motion is accepted, and the accept decision carries
+        no authorization claim.
+        """
+        snapshot = self.controller.read_safety_state()
+        accepted = self.controller.request_motion(
+            self.request(snapshot["state_token"])
+        )
+        self.assertEqual(accepted["controller_decision"], "accepted")
+        self.assertEqual(accepted["execution_status"], "completed")
+        # The accept result says nothing about scope, declared purpose or which
+        # agent issued the request. None of these are inputs the controller can
+        # see, which is exactly why physical safety cannot stand in for trust.
+        for authorization_field in ("workflow_id", "agent_id", "declared_scope"):
+            self.assertNotIn(authorization_field, accepted)
+
 
 if __name__ == "__main__":
     unittest.main()