Agent Harness Lab

Companion repository for the article: Before You Add an Agent, Build the Harness.
Article link: coming soon.

A minimal AI decision harness that demonstrates a critical architectural principle:

The LLM may recommend an action, but the system must validate, constrain, route, and audit the final decision.

This version does not call a real LLM. The classifier is mocked on purpose so the harness architecture is visible without API keys, provider-specific code, or model variability.

---

What This Project Demonstrates

This project implements a small, deterministic decision pipeline for infrastructure or operational requests. It shows how an organization can safely integrate LLM-based classification without granting the model final authority.

The harness enforces a strict separation of concerns:

LLM classifies.
Schema validates.
Policy constrains.
Router decides.
Audit records.

The central behavior to observe is the override: the LLM may recommend auto_approve, but the harness can still escalate to human_review or reject based on deterministic policy rules.

Request: REQ-001
LLM recommendation: auto_approve
Final decision: human_review
Overrode LLM: true
Audit written to: audits/REQ-001.json

Why the LLM Does Not Make the Final Decision

Large language models are probabilistic tools. They can hallucinate, be jailbroken, miss subtle context, or simply recommend actions that violate organizational policy. Treating an LLM as the final authority turns probabilistic output into unaccountable automation.

Instead, this harness treats the LLM as a classifier and recommender only. Its output is:

1. Validated against a strict schema (Zod).
2. Evaluated by deterministic policy rules.
3. Potentially overridden by those rules.
4. Logged for accountability.

What the Harness Is

The harness is a lightweight CLI application that accepts a JSON request file and runs it through a fixed pipeline:

1. Load the request JSON.
2. Validate the request shape using Zod.
3. Classify the request using a mock LLM classifier.
4. Validate the classifier output using Zod.
5. Apply deterministic policy rules.
6. Route to a final decision.
7. Write an audit log under audits/<request_id>.json.
8. Print the final decision to stdout.

What This Is Not

• A production-ready agent platform.
• A workflow engine.
• A policy engine.
• An LLM framework.
• A replacement for human judgment.

It is a small architectural experiment that isolates one pattern:

Deterministic boundaries around probabilistic model output.

Branches

This repository is organized around two stages:

Branch	Purpose
main	Demonstrates the harness using a mock classifier. No API keys required.
with-llm	Replaces the mock classifier with a real LLM integration while preserving the same validation, policy, routing, and audit boundaries.

The important comparison is the diff between the branches: adding an LLM changes the classifier implementation, not the decision authority model.

Architectural Invariants

These constraints must remain true as the project evolves:

• The LLM is not the final decision-maker.
• All request inputs must be validated before processing.
• All LLM outputs must be validated before policy evaluation.
• Policy rules must be deterministic and inspectable.
• The final decision must be produced by routing logic, not by the LLM.
• Any override of the LLM recommendation must be explicit.
• Every run must produce an audit record.
• The system should favor safe escalation over unsafe automation.

Agent Guidance

This README is also context for coding agents working on the repo.

• Do not collapse the harness into a single prompt, model call, or function.
• Preserve the separation of responsibilities:
  • classifier
  • schema validation
  • policy evaluation
  • routing
  • audit logging
• Future real LLM integration must not bypass Zod validation.
• Future workflow engine integration must not move final decision authority into the LLM.
• Future policy changes should remain deterministic, testable, and auditable.
• Provider-specific LLM code should not leak into policy or routing logic.

Project Structure

agent-harness-lab/
├── examples/
│   ├── low-risk.json
│   ├── medium-risk.json
│   ├── high-risk.json
│   ├── forbidden.json
│   └── ambiguous.json
├── audits/
├── src/
│   ├── index.ts              # CLI entry point and orchestration
│   ├── schemas.ts            # Zod schemas for runtime validation
│   ├── types.ts              # Central TypeScript exports
│   ├── classify-request.ts   # Mock LLM classifier
│   ├── policy-check.ts       # Deterministic policy engine
│   ├── route-decision.ts     # Decision routing logic
│   └── audit-log.ts          # Audit log writer
├── package.json
├── tsconfig.json
└── README.md

Getting Started

Install dependencies

npm install

Run the examples

# Low risk — auto-approved
npm run review examples/low-risk.json

# Medium risk — classifier recommends human review
npm run review examples/medium-risk.json

# High risk — production + admin API
npm run review examples/high-risk.json

# Forbidden — policy forces rejection
npm run review examples/forbidden.json

# Ambiguous — low confidence triggers escalation
npm run review examples/ambiguous.json

The current branch uses a mock classifier, so no API keys or external services are required.

Build and typecheck

npm run typecheck
npm run build

Available Commands

Command	Description
npm run review <file>	Run the harness against a JSON request file
npm run typecheck	Run strict TypeScript type checking (no emit)
npm run build	Compile TypeScript to dist/

Example Output

Request: REQ-001
LLM recommendation: auto_approve
Final decision: human_review
Overrode LLM: true
Audit written to: audits/REQ-001.json

How the Policy Rules Work

Policy rules are deterministic, independent from the classifier, and organized by severity:

auto_approve < request_more_info < human_review < reject

Each rule may raise the severity, but no rule may lower it. The policy engine evaluates all rules and returns the highest severity required.

Current Rules

Rule	Trigger	Minimum Decision
production_environment	environment === "production"	human_review
public_exposure	Text mentions public exposure	human_review
admin_api	Text mentions admin API	human_review
disable_authentication	Text mentions disabling auth	reject
secrets_or_credentials	Text mentions secrets or credentials	human_review
sensitive_data	Text mentions customer data, PII, billing, etc.	human_review
broad_aws_permissions	Text mentions AdministratorAccess, wildcard, root access	human_review
low_confidence	Classifier confidence < 0.75	request_more_info

Override Behavior

The routeDecision function compares the LLM recommendation against the policy requirement and returns the more severe of the two. If the final decision differs from the LLM recommendation, the overrode_llm flag is set to true.

Audit Log Format

Every run produces a JSON audit log:

{
  "request": {},
  "llm_classification": {},
  "policy_decision": {
    "final_action": "human_review",
    "triggered_rules": ["production_environment", "admin_api"],
    "overrode_llm": true
  },
  "audit": {
    "timestamp": "2026-01-15T12:00:00.000Z",
    "status": "recorded"
  }
}

Future Evolution

This harness is intentionally minimal. It is an architectural experiment, not a framework. Here are natural evolution paths:

Real LLM Integration

Replace classifyRequest with a call to OpenAI, Anthropic, or a local model. The rest of the pipeline remains unchanged because the LLM output is still validated by Zod and constrained by policy.

Human Approval

Introduce a human-in-the-loop step when the final decision is human_review. This could be:

• A Slack notification to an on-call engineer.
• A GitHub issue or pull request review.
• A custom approval dashboard.

Durable Workflow Engine

For production use, replace the simple CLI script with a durable workflow engine:

• Inngest — event-driven, great for retries and scheduling.
• Restate — durable execution with stateful handlers.
• Temporal — battle-tested for long-running, fault-tolerant workflows.
• LittleHorse — modern workflow-as-code with strong typing.

These engines can pause for human approval, retry failed LLM calls, and make audit writes more reliable through durable execution and idempotency controls.

Richer Audit and Policy Management

• Store audit logs in S3 or a database instead of the local filesystem.
• Replace hardcoded policy rules with a dynamic policy engine (e.g., Open Policy Agent).
• Add fine-grained RBAC so different teams have different policy thresholds.
• Track policy rule versions and migrate historical audits.

License

MIT