diff --git a/docs/src/content/docs/guides/custom-otlp-attributes.md b/docs/src/content/docs/guides/custom-otlp-attributes.md index 942628e0de2..189dc55f434 100644 --- a/docs/src/content/docs/guides/custom-otlp-attributes.md +++ b/docs/src/content/docs/guides/custom-otlp-attributes.md @@ -105,14 +105,17 @@ Sanitization is applied to both the over-the-wire OTLP export and the local JSON ## Debugging without a live collector -Every span is always appended as a sanitized JSON line to `/tmp/gh-aw/otel.jsonl`, even when `OTEL_EXPORTER_OTLP_ENDPOINT` is not set. This file is included in the `firewall-audit-logs` artifact so you can inspect spans after the run: +Every span emitted by `logSpan` is always appended as a sanitized JSON line to `/tmp/gh-aw/otel.jsonl`, even when `OTEL_EXPORTER_OTLP_ENDPOINT` is not set. When OTLP is configured, Copilot CLI's own spans are written to `/tmp/gh-aw/copilot-otel.jsonl` and automatically forwarded to configured endpoints at the end of the run. Both files are included in the `agent` artifact when OTLP is enabled, so you can inspect spans after the run: ```bash -# Download firewall/telemetry artifacts for a run -gh aw logs --artifacts firewall +# Download agent artifacts for a run +gh aw logs --artifacts agent # Inspect spans emitted by your tool cat otel.jsonl | jq 'select(.resourceSpans[].scopeSpans[].spans[].name | startswith("my-tool"))' + +# Inspect Copilot CLI spans +cat copilot-otel.jsonl | jq '.resourceSpans' ``` ## Advanced: low-level API diff --git a/docs/src/content/docs/reference/artifacts.md b/docs/src/content/docs/reference/artifacts.md index 54ffacce306..d2885aa3c71 100644 --- a/docs/src/content/docs/reference/artifacts.md +++ b/docs/src/content/docs/reference/artifacts.md @@ -122,6 +122,8 @@ The unified `agent` artifact contains all agent job outputs. - Safe output data (`agent_output.json`) - GitHub API rate limit logs (`github_rate_limits.jsonl`) - Token usage summary (`agent_usage.json`) — aggregated totals only; per-request data is in `firewall-audit-logs` +- `otel.jsonl` — OTLP span mirror written by gh-aw's JavaScript span exporters (only present when `observability.otlp` is configured) +- `copilot-otel.jsonl` — OTLP spans emitted by Copilot CLI (only present when `observability.otlp` is configured) ## `activation` diff --git a/docs/src/content/docs/reference/frontmatter.md b/docs/src/content/docs/reference/frontmatter.md index 40be35aac02..dc39c2e7332 100644 --- a/docs/src/content/docs/reference/frontmatter.md +++ b/docs/src/content/docs/reference/frontmatter.md @@ -849,12 +849,54 @@ observability: | Field | Type | Description | |-------|------|-------------| -| `observability.otlp.endpoint` | string | OTLP/HTTP collector endpoint URL (e.g. `https://traces.example.com:4318`). Supports GitHub Actions expressions. When a static URL is provided, its hostname is automatically added to the network firewall allowlist. | -| `observability.otlp.headers` | map or string | HTTP headers sent with every OTLP export request. | +| `observability.otlp.endpoint` | string, object, or array | OTLP/HTTP collector endpoint URL. Accepts a plain URL string, a single `{url, headers}` object, or an array of `{url, headers}` objects for concurrent fan-out to multiple collectors. When a static URL is provided, its hostname is automatically added to the network firewall allowlist. | +| `observability.otlp.headers` | map or string | HTTP headers sent with every OTLP export request. Only applies when `endpoint` is a plain string; object and array endpoint entries carry their own per-endpoint headers. | + +### `observability.otlp.endpoint` + +The `endpoint` field accepts three forms: + +**String form** (backward-compatible) — a plain URL with optional top-level `headers`: + +```yaml wrap +observability: + otlp: + endpoint: ${{ secrets.OTLP_ENDPOINT }} + headers: + Authorization: ${{ secrets.OTLP_TOKEN }} +``` + +**Object form** — a single endpoint with per-endpoint headers: + +```yaml wrap +observability: + otlp: + endpoint: + url: ${{ secrets.OTLP_ENDPOINT }} + headers: + Authorization: ${{ secrets.OTLP_TOKEN }} + X-Tenant: acme +``` + +**Array form** — multiple endpoints for concurrent fan-out: + +```yaml wrap +observability: + otlp: + endpoint: + - url: ${{ secrets.OTLP_ENDPOINT_PRIMARY }} + headers: + Authorization: ${{ secrets.OTLP_TOKEN_PRIMARY }} + - url: ${{ secrets.OTLP_ENDPOINT_BACKUP }} + headers: + Authorization: ${{ secrets.OTLP_TOKEN_BACKUP }} +``` + +When using the array form, spans are sent to all endpoints concurrently. A failure on one endpoint does not prevent export to others. ### `observability.otlp.headers` -The `headers` field accepts two forms: +The `headers` field accepts two forms (applies to the string endpoint form only): **Map form** — define each header as a key/value pair: @@ -876,6 +918,21 @@ observability: headers: "Authorization=${{ secrets.OTLP_TOKEN }},X-Tenant=acme" ``` +### Injected environment variables + +When `observability.otlp` is configured, the following environment variables are automatically injected into every step of the generated workflow: + +| Variable | Description | +|----------|-------------| +| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP collector URL (first endpoint, for backward compatibility with the MCP gateway and third-party tools). | +| `OTEL_EXPORTER_OTLP_HEADERS` | Comma-separated `key=value` headers for the first endpoint. Set only when headers are configured. | +| `OTEL_SERVICE_NAME` | Always `gh-aw`. | +| `GH_AW_OTLP_ENDPOINTS` | JSON-encoded array of all endpoint entries (`[{"url":"...","headers":"..."}]`). Used by JavaScript action scripts to fan out spans to multiple endpoints. | +| `COPILOT_OTEL_FILE_EXPORTER_PATH` | Path where Copilot CLI writes its own OTLP spans (`/tmp/gh-aw/copilot-otel.jsonl`). Copilot CLI detects this variable and writes its traces here; gh-aw forwards these traces to configured endpoints at the end of each run. | + +> [!NOTE] +> `GH_AW_OTLP_ENDPOINTS` is the primary variable used by gh-aw's JavaScript span exporters. `OTEL_EXPORTER_OTLP_ENDPOINT` is retained for backward compatibility only. + ### Agent span attributes The agent span (`gh-aw.agent.agent`) uses [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) and is emitted as a `SPAN_KIND_CLIENT` span. The following attributes are set on the agent span: