Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
244 lines (185 loc) · 5.81 KB

ARCHITECTURE.md

File metadata and controls

244 lines (185 loc) · 5.81 KB
Raw

Architecture

Goal

AgentOps Workbench ingests coding-agent session artifacts, normalizes them into a stable local event model, analyzes risk and evidence, stores the result in SQLite, and emits reviewable reports.

System Context

agent transcript / hook events / exported logs
        |
        v
ingestion adapter
        |
        v
normalized event stream
        |
        +--> SQLite local store
        |
        +--> analyzers
        |      - risk flags
        |      - verification evidence
        |      - loop/stall detection
        |      - cost/token extraction
        |
        +--> outputs
               - Markdown session report
               - PR/repo report
               - local dashboard/API
               - future OTLP/OpenTelemetry export

Core Modules

CLI

The CLI is the primary stable interface in v1.0.0.

Implemented commands include:

  • agentops audit <session.jsonl --quiet>
  • agentops adapters
  • agentops sessions
  • agentops look --session <id|latest>
  • agentops save report --session latest
  • agentops save pr --session latest
  • agentops save json --session latest
  • agentops save json --format openinference --session latest
  • agentops check --session latest
  • agentops config --check
  • agentops open --check
  • agentops open
  • agentops scan-publication

Ingestion Adapters

Adapters parse runner-specific artifacts into the normalized event model.

Implemented adapters:

  • agentops-jsonl: canonical agentops.event.v1 JSONL — any sanitized export (Claude Code, Codex, PAI/KAI, synthetic fixtures); provenance is read from each record's source field.
  • claude-code-stream-json: native Claude Code stream JSON from explicit CLI capture.
  • codex-exec-jsonl: native Codex exec JSONL from explicit CLI capture.
  • forensic-text: best-effort plain terminal/chat transcript import.

Candidate future adapters:

  • Hook-envelope adapter for bounded local capture.
  • OTLP/OpenTelemetry export adapter.

Adapter rules:

  • No analyzer logic inside adapters.
  • Preserve raw event JSON only when safe or explicitly redacted.
  • Attach source metadata without exposing local paths in public reports.

Normalized Event Model

Minimum event types:

  • message
  • plan
  • tool_call
  • command
  • file_read
  • file_write
  • file_edit
  • test_run
  • error
  • retry
  • final_response

Core event attributes:

  • session id
  • sequence index
  • timestamp when available
  • event type
  • role
  • summary
  • source adapter
  • raw payload hash
  • optional redacted raw payload

SQLite Store

The local database stores normalized sessions and analysis outputs.

Core tables:

  • sessions
  • events
  • tool_calls
  • commands
  • file_changes
  • checks
  • risk_flags
  • reports
  • costs

MVP already has:

  • sessions
  • events
  • commands
  • file_changes
  • risk_flags

Analyzer Pipeline

Analyzer passes should be deterministic and composable:

  • command risk analyzer
  • file risk analyzer
  • verification analyzer
  • final-claim analyzer
  • secret-looking value analyzer
  • churn analyzer
  • retry/loop analyzer
  • generated-file analyzer
  • cost/token analyzer

Each analyzer should emit structured findings:

  • severity
  • category
  • message
  • related event id
  • confidence
  • remediation hint

Report Generator

Markdown remains the primary output because it can be reviewed in terminals, PR comments, GitHub issues, and docs.

Report sections:

  • session summary
  • timeline
  • files touched
  • commands run
  • tests and verification evidence
  • risk flags
  • stalls/retries/loops
  • cost/token summary when available
  • final outcome assessment
  • appendices for raw metadata hashes

Dashboard

The dashboard is a local layer over the same SQLite store used by the CLI. Its browser endpoints are implementation details, not a remote API contract.

Dashboard views:

  • session list
  • timeline
  • file/command drilldown
  • risk/evidence cards
  • tool/MCP map
  • trend view across sessions

PAI Integration Boundary

PAI should be treated as an optional local integration, not a dependency.

Public architecture language:

  • AgentOps Workbench can ingest sanitized local agent-session artifacts produced by a personal assistant or agent runtime.
  • A PAI adapter may read exported session artifacts or hook envelopes only when the user explicitly opts in.
  • The workbench must never require access to private memory stores, credential stores, relationship stores, raw learning stores, security stores, identity stores, or failure-capture stores.

Integration modes:

  1. Post-hoc import: PAI or another agent exports a sanitized session artifact; AgentOps ingests it.
  2. Hook mirror: PAI emits bounded event envelopes into an AgentOps-compatible JSONL stream.
  3. Report handoff: AgentOps produces Markdown/JSON reports that PAI can summarize later.

Preferred initial mode: post-hoc import.

Reasons:

  • least invasive
  • easiest to sanitize
  • easy to explain publicly
  • does not couple product viability to a private PAI deployment

Public Data Boundary

Never commit:

  • raw personal transcripts
  • private PAI memory
  • local absolute paths
  • account names
  • email addresses
  • API keys or tokens
  • raw command output from real systems
  • private repo names unless intentionally public
  • screenshots containing private terminal context

Commit only:

  • synthetic fixtures
  • redacted fixtures
  • schema examples
  • public docs
  • generated example reports from synthetic data

Future OpenTelemetry Alignment

AgentOps Workbench should map internal event names to OpenTelemetry/GenAI concepts where practical:

  • session -> trace/session concept
  • tool call -> span/event
  • command -> custom span/event
  • file change -> repo-domain event
  • risk flag -> event/attribute or separate finding
  • token/cost -> metrics

MVP does not need OTLP export, but the schema should keep stable identifiers and timestamps so export can be added later.