Hermes support

.cargo/audit.toml

@@ -65,4 +65,20 @@ ignore = [
     # behavior; no runtime/network exposure beyond what rustls itself provides.
     # TODO: drop this ignore once rustls-native-certs / tokio-rustls roll forward.
     "RUSTSEC-2025-0134",
+    # RUSTSEC-2026-0173 — `proc-macro-error2 2.0.1` is unmaintained
+    # https://rustsec.org/advisories/RUSTSEC-2026-0173
+    #
+    # Severity: informational ("unmaintained"), NOT a vulnerability.
+    # Source: transitive build-time chain (controller-only):
+    #   oci-client 0.16.1 / oci-client 0.15.0 → oci-spec → getset 0.1.6 (proc-macro)
+    #     → proc-macro-error2 2.0.1
+    # Same rationale as RUSTSEC-2024-0370 above — this is a proc-macro crate
+    # that runs at compile time inside rustc, not at runtime in the controller
+    # binary. There is no runtime attack surface from an unmaintained
+    # proc-macro crate beyond the build toolchain itself. The advisory was
+    # published 2026-06-07 (one day before this commit); no upstream
+    # replacement exists yet — `getset` would need to roll forward.
+    # TODO: drop this ignore when `getset` switches off `proc-macro-error2`
+    # or when oci-client upgrades past `getset 0.1.6`.
+    "RUSTSEC-2026-0173",
 ]

.github/skills/agt-e2e-encryption/SKILL.md

@@ -7,6 +7,8 @@ description: "Kars AGT E2E encryption skill — how the Signal Protocol inter-ag
 ## Overview
 Kars agents communicate via the official Microsoft Agent Governance Toolkit (AGT) — a decentralized, E2E encrypted messaging protocol using Signal Protocol (X3DH + Double Ratchet). Transport is provided through `@microsoft/agent-governance-sdk` (upstream) wrapped by the in-repo `@kars/mesh` provider, selected at runtime via `createMeshTransport()`.
 
+OpenClaw runtimes go through the TypeScript SDK; **Hermes** runtimes go through `runtimes/agt-mesh-python/` (kars-agt-mesh), a Python AGT MeshClient at byte-for-byte wire-format parity with the TS SDK. The same relay/registry pair serves both. Cross-runtime bidi (`OpenClaw ↔ Hermes`) is proven on AKS and local kind — see `tests/e2e/interop/`.
+
 > **History note:** Earlier releases used a vendored fork of `@agentmesh/sdk` with 8+ local patches under `vendor/`. All upstream-relevant fixes have landed in AGT (PRs through the toolkit's 3.x line) and the vendored tree has been removed. The bug table below is preserved as a regression checklist — if any of these symptoms reappear, treat as a regression in the official SDK.
 
 ## Protocol Flow
@@ -90,3 +92,69 @@ not the WS relay.
 per known peer plus a 30 s heartbeat tick, but only receives back
 the relay's KNOCK-ack until a real bidirectional conversation
 starts. Counters live in the router process and **reset on pod restart**.
+
+---
+
+## Cross-runtime mesh (Hermes ↔ OpenClaw)
+
+### Wire format
+
+| Component | TS SDK | Python `kars_agt_mesh` |
+|---|---|---|
+| MESSAGE frame | `{v, type:"message", from, to, id, ts, header:{dh, pn, n}, ciphertext}` (std base64, NOT urlsafe) | identical |
+| KNOCK establishment | `{ik, ek, otk?}` — **short** keys | identical |
+| Plaintext payload | `JSON.parse(plaintext)` hardcoded on receive | sender JSON-wraps; receiver unwraps (`_payload_to_wire_bytes` / `_wire_bytes_to_payload`) |
+| AAD inside Double Ratchet | `caller_ad || x3dh_ad` where `caller_ad = "${sender_did}|${receiver_did}"` and `x3dh_ad = IK_init || IK_resp` (32 + 32 = 64) | identical |
+| Header serialization | `dh(32) || pcl(u32 BE) || mn(u32 BE)` = 40 bytes | identical |
+| HKDF (X3DH) | `HKDF(salt=zero[32], info="AgentMesh_X3DH_v1", IKM=F||dhConcat, len=32)`, F = `0xFF × 32` | identical |
+| Connect-frame POP | std-base64 (NOT urlsafe) `public_key` + Ed25519 sig over timestamp | identical |
+
+Both sides are pinned to the same upstream commit via `vendor/agt/pin.json::sha` — a single source of truth across TS, Python, and Rust.
+
+### Prekey-clobber guard (DO NOT BYPASS)
+
+`MeshClient.connect()` takes an exclusive `fcntl.flock` on `<identity_dir>/.mesh-prekeys.lock` (`runtimes/agt-mesh-python/src/kars_agt_mesh/client.py::_acquire_prekey_writer_lock`). A second Python process attempting to start a MeshClient for the same identity raises `MeshTransportError` naming the holder PID.
+
+**DO NOT** debug a live Hermes pod by running `kubectl exec ... python3 -c "from kars_runtime_hermes.plugin import mesh; mesh._get_or_init_client(); ..."`. The secondary process would (before the guard) generate fresh X3DH key material and `PUT` it to the registry, clobbering the daemon's bundle and causing silent `Decrypt failed for did:mesh:...` log entries with no traceback. Cost the team several hours of debugging in 2026-06; see [`docs/internal/security-audits/2026-06-06-cross-runtime-mesh-aks.md`](../../docs/internal/security-audits/2026-06-06-cross-runtime-mesh-aks.md).
+
+**Correct ways to inspect a live daemon:**
+
+```bash
+# Identity / DID
+kubectl exec <pod> -c agent -- cat /sandbox/.hermes/.agt/identity.json | jq
+
+# Operator trust store (peers seen by the daemon's mesh_worker)
+ADMIN=$(kubectl get secret -n kars-<sb> admin-token -o jsonpath='{.data.token}' | base64 -d)
+kubectl exec <pod> -c agent -- sh -c "curl -sS -H 'Authorization: Bearer $ADMIN' http://127.0.0.1:8443/agt/trust"
+
+# Daemon log
+kubectl logs <pod> -c agent
+```
+
+### AKS-specific Entra requirements
+
+Hermes pods on AKS require **two** RBAC arms for the mesh to bring up the verified tier:
+
+1. **The pod's workload-identity client (Entra Agent App auto-provisioned per sandbox)** — must have `Azure AI User` on the resource group. Auto-granted by the controller when you set `KarsAuthConfig.spec.foundryRbac` in the sandbox namespace. The CRD field landed in commit `496cc92`.
+2. **The Foundry project's MI** — needs `Azure AI User` on the resource group for Memory Store operations. This is a one-time Portal step per project (see the `Foundry Memory Store Auth` section in the repo's CLAUDE.md).
+
+The entrypoint exchanges the projected SA token for an Entra Agent App token (audience `<app-id>/.default`) and POSTs it to `/agt/registry/v1/registry/verify`. Success surfaces as `tier=verified, verified_app_id=<guid>` on the operator panel.
+
+### `kars push --only runtime-hermes` is hermetic
+
+Out of the box (no manual `git clone`, no `bash runtimes/build-agt-wheels.sh`):
+
+```bash
+kars push --only runtime-hermes --apply
+```
+
+The CLI auto-clones `microsoft/agent-governance-toolkit@kars-sdk-pop-signing@<pin sha>` (via `cli/src/lib/agt-bootstrap.ts::ensureAgtRepo`) and builds `runtimes/wheels/*.whl` (via `ensureAgtWheels`) before docker build. Cached by `runtimes/wheels/.agt-sha` so re-runs are no-ops.
+
+### End-to-end harnesses
+
+| Scope | Harness | Expected proof |
+|---|---|---|
+| Local kind, Hermes ↔ OpenClaw bidi | `tests/e2e/interop/hermes_openclaw_bidi.sh` | `score ≥ 500, status=delivered_and_replied` |
+| AKS production, four scenarios | `tests/e2e/interop/aks_full_suite.sh` | 4/4 PASS (single, inter-agent, multi-fanout, Entra Verified) |
+
+If either regresses, the first thing to suspect is the AGT pin — `vendor/agt/pin.json::sha` must match upstream `microsoft/agent-governance-toolkit:kars-sdk-pop-signing`. Pre-existing image layers may have a stale SDK; `kars push --only runtime-hermes --apply` (or `runtime-*` per affected runtime) refreshes them.

.github/workflows/ci.yml

@@ -294,6 +294,53 @@ jobs:
           python -m ruff check .
           python -m pytest
 
+  hermes-runtime:
+    name: Hermes Runtime (Python) Build & Test
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: "3.12"
+      - name: Install kars-agt-mesh (sibling dep) + hermes (+dev extras)
+        run: |
+          python -m pip install --upgrade pip
+          # hermes plugin imports kars_agt_mesh at module load, so the sibling
+          # in-repo package must be installed first; matches the order in
+          # sandbox-images/hermes/Dockerfile.
+          pip install -e runtimes/agt-mesh-python
+          pip install -e "runtimes/hermes[dev]"
+      - name: Run hermes ruff lint
+        run: |
+          cd runtimes/hermes
+          python -m ruff check .
+      - name: Run hermes pytest
+        run: |
+          cd runtimes/hermes
+          python -m pytest -v
+
+  kars-agt-mesh-python:
+    name: kars-agt-mesh (Python AGT mesh transport) Test
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: "3.12"
+      - name: Install kars-agt-mesh (+test extras)
+        run: |
+          cd runtimes/agt-mesh-python
+          python -m pip install --upgrade pip
+          pip install -e ".[test]"
+      - name: Run kars_agt_mesh ruff lint
+        run: |
+          cd runtimes/agt-mesh-python
+          python -m ruff check .
+      - name: Run kars_agt_mesh pytest
+        run: |
+          cd runtimes/agt-mesh-python
+          python -m pytest -v
+
   bicep-validate:
     name: Bicep Validation
     runs-on: ubuntu-latest

.gitignore

@@ -420,6 +420,10 @@ FodyWeavers.xsd
 # Rust
 target/
 target-linux-*/
+# Cross-compile artifacts produced by docker run -v $PWD:/work …
+# (e.g. `docker run … rust:1.89 cargo build` with CARGO_HOME=/work/.cargo-docker
+# to keep the host's ~/.cargo cache unpolluted by container-arch builds).
+.cargo-docker/
 Cargo.lock
 # Vendor lockfiles needed for reproducible Docker builds
 !vendor/*/Cargo.lock

Cargo.toml

@@ -40,6 +40,13 @@ futures = "0.3"
 # HTTP
 axum = { version = "0.8", features = ["ws"] }
 reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json", "stream", "gzip", "brotli", "deflate"] }
+# rustls 0.23 with aws-lc-rs is the only provider in our dep graph
+# (transitively via reqwest + oci-client). We declare it as a direct
+# workspace dep so the router's main() can install the provider
+# explicitly — otherwise kube-client's first TLS handshake panics
+# with "Could not automatically determine the process-level
+# CryptoProvider" when multiple feature flags resolve.
+rustls = { version = "0.23", default-features = false, features = ["aws-lc-rs"] }
 tower = { version = "0.5", features = ["limit"] }
 tower-http = { version = "0.6", features = ["trace", "cors"] }
 hyper = "1"
@@ -101,24 +108,27 @@ lto = true
 codegen-units = 1
 strip = true
 
-# ─── Temporary git override for AGT pre-release (POP-aware SDK) ───
-# Pinned to pallakatos/agent-governance-toolkit@bdea1097 — fork-branch
-# that adds the TypeScript SDK side of proof-of-possession registration
-# and connect-frame signing. Upstream PR:
-# https://github.com/microsoft/agent-governance-toolkit/pull/2772
+# ─── Temporary git override for AGT pre-release fixes ───
+# Pinned to microsoft/agent-governance-toolkit@3322175d on the
+# `kars-sdk-pop-signing` feature branch (upstream repo, not a fork).
+# This branch carries two pre-release fixes kars depends on:
+#   1. TypeScript SDK proof-of-possession registration + connect-frame
+#      signing (upstream PR #2772, in review).
+#   2. X3DH KDF spec-compliance per Signal §2.2 (F prefix in IKM,
+#      zero salt — separate upstream PR pending).
 #
-# The Rust agentmesh crate is unchanged between upstream `bae5de3` and
-# our `bdea1097` (only the TypeScript SDK was touched); we still point
-# the Rust [patch.crates-io] override at the fork SHA so the workspace
-# stays at one consistent toolkit revision and `ci/check-agt-released.sh`
+# The Rust agentmesh crate is unchanged between upstream main and our
+# pin (only the TypeScript SDK was touched); we still point the Rust
+# [patch.crates-io] override at the branch SHA so the workspace stays
+# at one consistent toolkit revision and `ci/check-agt-released.sh`
 # can track a single SHA.
 #
 # Remove this whole [patch.crates-io] block when:
-#   1. PR #2772 merges to microsoft/agent-governance-toolkit main, AND
-#   2. AGT cuts a release containing it (track via the daily check).
+#   1. Both PRs merge to microsoft/agent-governance-toolkit main, AND
+#   2. AGT cuts a release containing them (track via the daily check).
 # At that point also drop the `vendor/agt/microsoft-agent-governance-sdk-*.tgz`
 # tarball + the file: dep in mesh-plugin/package.json and switch to the
 # published npm release.
 [patch.crates-io]
-agentmesh = { git = "https://github.com/pallakatos/agent-governance-toolkit", rev = "bdea10970f641fa695bb6dce7f8cee425f8aebd6", package = "agentmesh" }
-agentmesh-mcp = { git = "https://github.com/pallakatos/agent-governance-toolkit", rev = "bdea10970f641fa695bb6dce7f8cee425f8aebd6", package = "agentmesh-mcp" }
+agentmesh = { git = "https://github.com/microsoft/agent-governance-toolkit", rev = "3322175d88baf61e8ceab8e392e29fa2bf9b580a", package = "agentmesh" }
+agentmesh-mcp = { git = "https://github.com/microsoft/agent-governance-toolkit", rev = "3322175d88baf61e8ceab8e392e29fa2bf9b580a", package = "agentmesh-mcp" }

README.md

@@ -46,7 +46,7 @@ It is built for three audiences:
 - **A prompt-injected agent cannot leak your Azure credentials** — because the agent process runs under a different UID than the Rust router that holds them, on a different network, and never sees the bearer token. (iptables + NetworkPolicy back this up; they are not the primary control.)
 - **Your security and platform teams review YAML, not Python.** Approval gates, rate limits, tool allowlists, content-safety floors, token budgets, and trust topology are declarative Kubernetes resources — commit them to a repo, reconcile with Argo / Flux, audit with `git log`. No out-of-band config store, no per-agent code review for policy.
 - **Two agents that talk cannot be eavesdropped by you, by us, or by the relay.** Agent-to-agent messaging is end-to-end encrypted with Signal Protocol (X3DH + Double Ratchet) and KNOCK trust gating. The relay sees only ciphertext.
-- **You are not locked to one runtime or one model provider.** Seven first-class runtimes (OpenClaw, OpenAI Agents SDK, Microsoft Agent Framework, LangGraph in Python and TypeScript, Anthropic, Pydantic-AI) plus BYO. GitHub Copilot, Azure AI Foundry, Azure OpenAI, and GitHub Models as backends — switching is a one-field CRD change, not a re-architecture. Native Anthropic-shape passthrough for Claude.
+- **You are not locked to one runtime or one model provider.** Eight first-class runtimes (OpenClaw, Hermes, OpenAI Agents SDK, Microsoft Agent Framework, LangGraph in Python and TypeScript, Anthropic, Pydantic-AI) plus BYO. GitHub Copilot, Azure AI Foundry, Azure OpenAI, and GitHub Models as backends — switching is a one-field CRD change, not a re-architecture. Native Anthropic-shape passthrough for Claude.
 - **What runs on your laptop is what runs in production.** `kars dev` and `kars up` share the same router binary, the same audit chain, the same governance profile. The dev-to-prod jump is one CLI command — no separate stack to learn or debug.
 
 ---
@@ -258,6 +258,7 @@ You pick the runtime via `KarsSandbox.spec.runtime.kind`. The router, governance
 | Runtime | Language | Image dir | Status |
 |---|---|---|---|
 | **OpenClaw** (default) | Python | `sandbox-images/openclaw/` | ✅ |
+| **Hermes** (Nous Research) | Python | `sandbox-images/hermes/` | ✅ |
 | **OpenAI Agents SDK** | Python | `sandbox-images/openai-agents/` | ✅ |
 | **Microsoft Agent Framework** | Python | `sandbox-images/maf-python/` | ✅ (`.NET` deferred) |
 | **LangGraph** | Python | `sandbox-images/langgraph/` | ✅ |