Skip to content

mvp-evaluator-guide.md

Latest commit

 

History

History
248 lines (198 loc) · 8.19 KB

mvp-evaluator-guide.md

File metadata and controls

248 lines (198 loc) · 8.19 KB
Raw

Ardur MVP Evaluator Guide

Use this source-checkout guide to evaluate Ardur's authenticated Docker demo: a SPIRE-backed local proxy that applies mission policy before tool execution and returns signed session evidence.

For the provider-free, no-bearer first-run path, use the No-Key MVP Demo instead. The relaxed auth mode in that guide is deliberately loopback-only and temporary.

Start the authenticated demo

From the repository root, configure a fresh local bearer token and start the stack in terminal 1:

export ARDUR_API_TOKEN="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
make demo

Wait until docker compose ps reports the SPIRE server, SPIRE agent, proxy, and hub as healthy. Keep terminal 1 running. make demo-down stops the stack and removes its named volumes after the walkthrough.

Run the complete lifecycle

Paste this entire block into terminal 2 from the same repository root. If ARDUR_API_TOKEN is not already exported there, the block reads the configured token from the running proxy container. It never prints the bearer value or places it directly in curl's argument list.

(
set -euo pipefail

PROXY_URL="${ARDUR_PROXY_URL:-https://localhost:${ARDUR_PROXY_PORT:-8443}}"
if [[ -z "${ARDUR_API_TOKEN:-}" ]]; then
  ARDUR_API_TOKEN="$(docker compose exec -T proxy sh -c 'printf %s "$VIBAP_API_TOKEN"')"
fi
if [[ -z "$ARDUR_API_TOKEN" || "$ARDUR_API_TOKEN" == *$'\n'* || "$ARDUR_API_TOKEN" == *$'\r'* ]]; then
  echo "No configured proxy token found. Start make demo with ARDUR_API_TOKEN set." >&2
  exit 1
fi

umask 077
AUTH_HEADER_FILE="$(mktemp "${TMPDIR:-/tmp}/ardur-evaluator-auth.XXXXXX")"
REQUEST_BODY_FILE="$(mktemp "${TMPDIR:-/tmp}/ardur-evaluator-body.XXXXXX")"
cleanup() {
  rm -f "$AUTH_HEADER_FILE" "$REQUEST_BODY_FILE"
}
trap cleanup EXIT
printf 'Authorization: Bearer %s\n' "$ARDUR_API_TOKEN" > "$AUTH_HEADER_FILE"

curl_public() {
  curl --insecure --silent --show-error --fail "$@"
}

curl_auth() {
  curl --insecure --silent --show-error --fail --header "@$AUTH_HEADER_FILE" "$@"
}

post_json() {
  local path="$1"
  local payload="$2"
  printf '%s' "$payload" > "$REQUEST_BODY_FILE"
  curl_auth \
    --request POST \
    --header 'Content-Type: application/json' \
    --data-binary "@$REQUEST_BODY_FILE" \
    "$PROXY_URL$path"
}

json_string() {
  python3 -c '
import json
import sys

value = json.load(sys.stdin).get(sys.argv[1])
assert isinstance(value, str) and value, value
print(value, end="")
' "$1"
}

json_value() {
  python3 -c '
import json
import sys

value = json.load(sys.stdin).get(sys.argv[1])
assert value is not None, value
print(value, end="")
' "$1"
}

json_object_with_stdin() {
  python3 -c '
import json
import sys

print(json.dumps({sys.argv[1]: sys.stdin.read()}), end="")
' "$1"
}

json_evaluate() {
  python3 -c '
import json
import sys

print(json.dumps({
    "session_id": sys.stdin.read(),
    "tool_name": sys.argv[1],
    "arguments": {"path": "/tmp/ardur-evaluator.txt"},
}), end="")
' "$1"
}

HEALTH_RESPONSE="$(curl_public "$PROXY_URL/health")"
test "$(printf '%s' "$HEALTH_RESPONSE" | json_value status)" = "ok"
echo "health=ok"

MISSION_PAYLOAD='{"mission":{"agent_id":"evaluator-guide","mission":"evaluate the governance proxy","allowed_tools":["read_file","delete_file"],"forbidden_tools":["delete_file"],"resource_scope":["**"],"max_tool_calls":4}}'
ISSUE_RESPONSE="$(post_json /issue "$MISSION_PAYLOAD")"
PASSPORT="$(printf '%s' "$ISSUE_RESPONSE" | json_string token)"
echo "issue=passport-created"

START_PAYLOAD="$(printf '%s' "$PASSPORT" | json_object_with_stdin token)"
START_RESPONSE="$(post_json /session/start "$START_PAYLOAD")"
SESSION_ID="$(printf '%s' "$START_RESPONSE" | json_string session_id)"
echo "session=started"

PERMIT_PAYLOAD="$(printf '%s' "$SESSION_ID" | json_evaluate read_file)"
PERMIT_RESPONSE="$(post_json /evaluate "$PERMIT_PAYLOAD")"
PERMIT_DECISION="$(printf '%s' "$PERMIT_RESPONSE" | json_string decision)"
test "$PERMIT_DECISION" = "PERMIT"
echo "read_file=$PERMIT_DECISION"

DENY_PAYLOAD="$(printf '%s' "$SESSION_ID" | json_evaluate delete_file)"
DENY_RESPONSE="$(post_json /evaluate "$DENY_PAYLOAD")"
DENY_DECISION="$(printf '%s' "$DENY_RESPONSE" | json_string decision)"
test "$DENY_DECISION" = "DENY"
echo "delete_file=$DENY_DECISION"

SESSION_PAYLOAD="$(printf '%s' "$SESSION_ID" | json_object_with_stdin session_id)"
ATTEST_RESPONSE="$(post_json /attest "$SESSION_PAYLOAD")"
ATTESTATION_TOKEN="$(printf '%s' "$ATTEST_RESPONSE" | json_string token)"
test -n "$ATTESTATION_TOKEN"
echo "attest=signed-token-created"

END_RESPONSE="$(post_json /session/end "$SESSION_PAYLOAD")"
END_ATTESTATION="$(printf '%s' "$END_RESPONSE" | json_string attestation_token)"
test -n "$END_ATTESTATION"
echo "session=ended"

METRICS_RESPONSE="$(curl_auth "$PROXY_URL/metrics")"
[[ "$METRICS_RESPONSE" == *"ardur_"* ]]
echo "metrics=prometheus-ok"
)

Expected output:

health=ok
issue=passport-created
session=started
read_file=PERMIT
delete_file=DENY
attest=signed-token-created
session=ended
metrics=prometheus-ok

Every curl call uses --fail, so an authentication or schema error makes the block exit non-zero instead of turning an HTTP 4xx body into a misleading pass. The same lifecycle and payload shapes are also exercised by scripts/verify-mvp.sh.

What the lifecycle proves

  • /issue signs a Mission Passport for a structured mission declaration.
  • /session/start binds a governed session to that passport.
  • /evaluate returns PERMIT for an allowed tool and DENY when a forbidden rule overlaps the allowlist; deny wins.
  • /attest and /session/end return signed behavioral-attestation JWTs.
  • /metrics is authenticated and exposes Prometheus-formatted Ardur metrics.

The walkthrough verifies that non-empty signed tokens are returned. For a local cryptographic signature-verification demonstration, run scripts/run-no-key-mvp-demo.py, which verifies the session-end token with its ephemeral public key before cleanup.

Architecture boundary

The local Compose stack contains a SPIRE server, a SPIRE agent, the governance proxy, and the Personal Hub. The proxy governs calls presented at its HTTP/tool boundary; it does not claim visibility into provider-hidden reasoning or every subprocess, filesystem, kernel, or network side effect caused below that boundary.

Kill switch

The emergency kill switch is an authenticated administrative control. It is not part of the copy-paste lifecycle above because it changes shared proxy state and would disrupt other evaluator sessions. The CLI uses ARDUR_PROXY_URL and ARDUR_API_TOKEN when the explicit flags are omitted:

export ARDUR_PROXY_URL="https://localhost:${ARDUR_PROXY_PORT:-8443}"
export ARDUR_API_TOKEN="$(docker compose exec -T proxy sh -c 'printf %s "$VIBAP_API_TOKEN"')"
ardur kill-switch
ardur kill-switch --deactivate

Health remains public while the switch is active. Authenticated governance operations fail closed until it is deactivated.

Stop the demo

In a third terminal, or after stopping the attached make demo process, run:

make demo-down

This removes the Compose containers, network, and named volumes for the project.

Known gaps

  • Capture boundary: Ardur governs at the tool-call boundary. See docs/coverage-map.md for current coverage and roadmap boundaries.
  • Development TLS: the local proxy uses a generated self-signed certificate, so the walkthrough uses curl's loopback-only --insecure mode. Do not carry that TLS policy to a remote deployment.
  • Single-user demo: the local stack is not a multi-tenant isolation model.
  • Token Status List scope: Credential-level Token Status List revocation checking lives in the Go credential verifier (go/pkg/credential). The Python path checks mission-level status lists (vibap.mission.mission_is_revoked) but does not yet implement the credential-level check.

Where to look next