Skip to content

Add opt-in OpenAI tracing for extract.ai debugging #1172

Description

@ebhills

Description

Add optional OpenAI tracing for deeper extract.ai debugging, following the logging, metadata, and retry work merged in PR #1171 (714008bc). Stored Responses logs and request metadata are already available; this issue covers the extraction workflow around those requests.

Desired Behavior

  • Provide an explicit tracing option for the Responses execution path, disabled by default.
  • Group activity by extraction invocation, including total/request timing, retries, errors, and local result-cache activity. Cache hits should be visible without causing another model request.
  • Correlate traces with the existing recipe/user diagnostic labels when available, keeping each invocation's trace context isolated across nested recipes and concurrent rows/runs.
  • Send traces to the intended OpenAI project using appropriately scoped export credentials. Confirm the supported tracing/export mechanism and authentication requirements before implementation; never overwrite shared SDK credentials to switch projects.
  • Bound exporter work and flush time for short-lived Lambda execution. Record export failures without masking the original extraction failure or delaying results beyond the configured export bound.
  • Keep tracing optional at installation/import time where practical. When disabled, do not initialize an exporter or send trace traffic.
  • Preserve extraction outputs, exceptions, retry decisions, cache keys/reuse, row ordering, and model-call counts whether tracing is enabled, disabled, or unavailable.
  • Exclude API keys, credentials, and unrelated recipe variables from trace data. Keep input/output content capture off by default and require a separate explicit option to include it.

Example current / desired

Current: an operator can inspect stored OpenAI Responses and their recipe/user labels, but has no trace connecting the Wrangles extraction invocation to its retries, timing, and cache reuse.

Desired: enabling tracing for one extraction produces a grouped view of that invocation and its request/cache activity. Cached rows remain cached, and an exporter timeout does not change the extraction result.

Acceptance criteria

  • Confirm the exporter/SDK approach, project credential routing, optional dependency requirements, and public option names before implementation.
  • Implement invocation-scoped instrumentation and bounded export/flush behavior.
  • Add sanitized, mocked tests for disabled behavior, cache hits, retry/error handling, nested/concurrent runs, credential isolation, exporter failures, and flush time limits.
  • Verify identical extraction results, original error propagation, cache identity/reuse, and model-call counts with tracing on and off.
  • Document the differences between store, request metadata, and workflow traces, with recipe/Python examples and content-capture controls.
  • Validate live export separately from mocked tests: confirm the intended OpenAI project receives traces and record a short-lived Lambda smoke test with runtime/version provenance.

Scope and deployment boundaries

Start with Responses tracing in extract.ai. Chat Completions tracing and tracing across an entire recipe are outside this issue. Preserve existing recipes and keep this work separate from PR #1171.

The deployed recipe Lambda is owned by Lambda-Recipes; identify any companion dependency or lifecycle change there before claiming deployed tracing support. Creating this issue does not start implementation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions