Skip to content

feat: Add live tracing mode for complete audit trail#300

Open
gizmo-dev wants to merge 3 commits intousestrix:mainfrom
gizmo-dev:feat/live-tracing-audit-log
Open

feat: Add live tracing mode for complete audit trail#300
gizmo-dev wants to merge 3 commits intousestrix:mainfrom
gizmo-dev:feat/live-tracing-audit-log

Conversation

@gizmo-dev
Copy link

Summary

Adds a live tracing mode that records a complete audit trail of runs in real-time, streaming structured events to disk as JSONL.

Closes #285

Features

  • Real-time JSONL tracing - Every event is written immediately to disk with timestamps and unique IDs
  • Comprehensive event coverage:
    • LLM requests/responses (with token counts)
    • Tool calls and results
    • Agent lifecycle (creation, completion, state changes)
    • Inter-agent messages
    • Vulnerability discoveries
  • Secret redaction - Optional automatic masking of API keys, tokens, passwords, and other sensitive data
  • CLI flags:
    • --trace / STRIX_TRACE=true - Enable tracing
    • --trace-output <path> - Custom output path
    • --redact-secrets / STRIX_REDACT_SECRETS=true - Enable secret redaction

Usage

# Enable tracing with secret redaction
strix --target ./app --trace --redact-secrets

# Custom output path
strix --target ./app --trace --trace-output ./audit/trace.jsonl

# Via environment variables
STRIX_TRACE=true STRIX_REDACT_SECRETS=true strix --target ./app

Files Changed

  • strix/telemetry/live_tracer.py - Core LiveTracer class
  • strix/telemetry/redactor.py - SecretRedactor for sensitive data masking
  • strix/llm/llm.py - LLM request/response tracing hooks
  • strix/tools/executor.py - Tool call/result tracing hooks
  • strix/agents/base_agent.py - Agent lifecycle tracing hooks
  • strix/config/config.py - New config options
  • strix/interface/main.py, cli.py, tui.py - CLI integration
  • tests/telemetry/test_live_tracer.py - LiveTracer tests
  • tests/telemetry/test_redactor.py - SecretRedactor tests

Testing

All tests pass:

poetry run pytest tests/telemetry/ -v

Example Trace Output

{"event_type":"trace_start","timestamp":"2026-02-03T...","run_id":"portal-gizmolab-io_0385","seq":0}
{"event_type":"agent_created","timestamp":"...","agent_id":"root","agent_name":"StrixAgent","seq":1}
{"event_type":"llm_request","timestamp":"...","model":"anthropic/claude-sonnet-4-5-20250514","seq":2}
{"event_type":"llm_response","timestamp":"...","model":"anthropic/claude-sonnet-4-5-20250514","tokens":{...},"seq":3}
{"event_type":"tool_call","timestamp":"...","tool_name":"thinking","seq":4}
...

Closes usestrix#285

This adds a comprehensive live tracing system that records a complete audit
trail of a run in real-time, streaming to disk as events happen. This is
essential for professional pentesting where evidence of actions taken is
required, and makes debugging agent behavior much easier.

## Features

- **JSONL trace format**: Clean, structured trace output with timestamps and IDs
- **CLI integration**: Enable via `--trace` flag or `STRIX_TRACE=1` env var
- **Secret redaction**: Optional redaction via `--redact-secrets` flag
- **Custom output path**: `--trace-output` to specify trace file location

## Events Captured

- `trace_start` / `trace_end`: Run lifecycle
- `llm_request` / `llm_response` / `llm_error`: All LLM interactions
- `tool_call` / `tool_result`: Every tool invocation with args and results
- `agent_created` / `agent_status_update`: Agent lifecycle events
- `message`: All agent/sub-agent messages
- `state_change`: Key state transitions
- `vulnerability_found`: Discovered vulnerabilities

## Usage

```bash
# Basic tracing
strix --target https://example.com --trace

# With secret redaction
strix --target https://example.com --trace --redact-secrets

# Custom output path
strix --target https://example.com --trace --trace-output ./audit.jsonl

# Via environment variables
STRIX_TRACE=1 STRIX_REDACT_SECRETS=1 strix --target https://example.com
```

## Files Added/Modified

- `strix/telemetry/live_tracer.py`: Core LiveTracer class
- `strix/telemetry/redactor.py`: SecretRedactor for sensitive data
- `strix/config/config.py`: Trace configuration options
- `strix/interface/main.py`: CLI argument handling
- `strix/llm/llm.py`: LLM request/response hooks
- `strix/tools/executor.py`: Tool call/result hooks
- `strix/agents/base_agent.py`: Agent lifecycle hooks
- Tests with 39 new test cases (all passing)
@greptile-apps
Copy link
Contributor

greptile-apps bot commented Feb 3, 2026

Greptile Overview

Greptile Summary

Added a comprehensive live tracing system that streams structured JSONL audit logs to disk in real-time, capturing complete execution traces of penetration testing runs.

Key Changes:

  • Implemented LiveTracer class with thread-safe JSONL streaming, monotonic sequence numbering, and structured event logging for LLM calls, tool executions, agent lifecycle, state changes, and vulnerability discoveries
  • Created SecretRedactor utility with pattern-based detection of API keys, tokens, passwords, and sensitive headers to enable safe trace sharing
  • Integrated tracing hooks across the codebase with proper timing, exception handling, and zero impact when tracing is disabled
  • Added CLI flags --trace, --trace-output, and --redact-secrets with corresponding environment variable support
  • Implemented proper cleanup in all exit paths (normal exit, signal handlers, exceptions) to ensure trace files are properly closed

Implementation Quality:

  • Thread-safe operations using locks for sequence generation and file writing
  • Defensive programming with try/except blocks around all trace calls to prevent failures from impacting core functionality
  • Smart truncation of large payloads (messages, results) to keep trace files manageable
  • Comprehensive test coverage with 701 lines of tests across two test files
  • Clean separation of concerns with dedicated modules for tracing and redaction

Confidence Score: 5/5

  • This PR is safe to merge with minimal risk - well-designed, thoroughly tested, and non-invasive
  • The implementation demonstrates excellent engineering practices: defensive exception handling prevents trace failures from impacting core operations, thread-safe operations avoid race conditions, comprehensive test coverage validates all functionality, and the feature is cleanly integrated with minimal coupling to existing code
  • No files require special attention

Important Files Changed

Filename Overview
strix/telemetry/live_tracer.py New 497-line core tracing module with comprehensive event logging, thread-safe operations, and context management
strix/telemetry/redactor.py New 209-line secret redaction utility with pattern matching for API keys, tokens, and sensitive headers
strix/llm/llm.py Added timing and trace hooks for LLM requests, responses, and errors with proper exception handling
strix/tools/executor.py Added timing and trace hooks for tool calls and results with duration tracking
strix/agents/base_agent.py Added agent lifecycle and message tracing with state change tracking across creation, completion, and messages
strix/interface/main.py Added CLI argument parsing, tracer initialization, and lifecycle management with proper cleanup
tests/telemetry/test_live_tracer.py Comprehensive 444-line test suite covering all tracing events, sequence ordering, truncation, and redaction
tests/telemetry/test_redactor.py Thorough 257-line test suite covering API keys, tokens, nested structures, and custom patterns

@dennisfurrer
Copy link

FINALLY! @gizmo-dev

The sandbox image's entrypoint no longer starts the tool server automatically.
This fixes the 'Tool server failed to start' error by:
1. Adding CAIDO_PORT env var required by the entrypoint
2. Starting the tool server via exec_run after container initialization
Adds a new --trace-verbose flag for non-interactive mode that outputs
human-readable trace events to the console in real-time, similar to
TUI output but in plain text format.

Features:
- New ConsoleTracer class for formatting trace events with Rich
- Shows LLM requests/responses, tool calls, agent events in real-time
- Uses emoji icons and color coding for different event types
- Truncates long output for readability
- Only works with --non-interactive mode

Usage:
  strix --target https://example.com --trace-verbose --non-interactive

Files changed:
- strix/telemetry/console_tracer.py - New console formatter
- strix/telemetry/live_tracer.py - Added verbose parameter
- strix/interface/main.py - Added --trace-verbose CLI flag
- strix/telemetry/__init__.py - Export ConsoleTracer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Detailed live tracing + full audit log of agent actions

2 participants