Add observability metrics

[ian-noaa]

This pull request introduces OpenTelemetry instrumentation and observability improvements to the SQS worker, along with related documentation updates and minor best practice clarifications. The main changes include initializing the OTel SDK, adding metrics and tracing to the SQS processing loop, and updating docs to reflect the new observability architecture.

Observability & Instrumentation:

• cmd/sqsworker/main.go, cmd/sqsworker/heartbeat.go: The SQS worker now initializes the OpenTelemetry SDK at startup, wires the OTel log bridge, and instruments AWS SDK clients for traces and metrics. Key processing steps (message receive, filter, process, delete) are tracked with metrics and spans.

Documentation Updates:

• docs/architecture.md, AGENTS.md: Updated project structure diagrams and descriptions to include the new internal/otel/ and pkg/telemetry/ packages. Added OpenTelemetry as a core dependency and described its role in metrics, tracing, and logging.
• docs/dev-guide.md: Updated the Go version requirement to 1.25+.

Minor Improvements:

• Improved error handling and logging to use context-aware logging and OTel status codes.
• Added explicit OTel shutdown and graceful termination logic to the SQS worker.

These changes lay the foundation for observability, making it easier to monitor, trace, and debug the SQS worker in production.

[Copilot]

Pull request overview

This pull request adds OpenTelemetry-based observability to the SQS worker path (traces, metrics, and slog-to-OTLP log export), and updates project documentation to reflect the new telemetry architecture and Go version requirement.

Changes:

• Introduces pkg/telemetry (OTel API-only) instruments and span naming, plus internal/otel SDK initialization (OTLP exporters + slog fan-out bridge).
• Instruments key execution paths (SQS polling/handling, S3 tarball download/extraction, stat parsing, DB upserts) with metrics and spans.
• Updates architecture/dev documentation to include the new observability layer and revised prerequisites.

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
pkg/telemetry/spans.go	Adds shared tracer and span name constants used across the worker pipeline.
pkg/telemetry/metrics.go	Declares counters/histograms for health and bad-data signals; provides InitMetrics.
pkg/storage/s3tarball.go	Adds context-aware logging plus S3 download duration + tarball extraction error metrics and spans.
pkg/core/statToCbRun.go	Propagates context into stat parsing; emits per-file success/error metrics and span status.
pkg/core/statFileToCbDocMetParser.go	Adds parse-line metrics (lines parsed/skipped, parse errors) and context-aware logging.
pkg/core/statFileToCbDocMetParser_test.go	Updates tests/benchmarks to match the new parseStatFileContent(ctx, ...) signature.
pkg/async/flushToDbAsync.go	Adds DB upsert duration + success/error counters and merge metrics in async upsert worker.
internal/otel/slogbridge.go	Adds slog handler fan-out to stdout JSON + OTel log bridge.
internal/otel/otel.go	Adds OTel SDK initialization (trace/metric/log providers, runtime metrics, shutdown).
cmd/sqsworker/main.go	Initializes OTel SDK, instruments polling/message handling, and adds graceful telemetry shutdown.
cmd/sqsworker/heartbeat.go	Adds success/error metrics for visibility timeout heartbeat extensions.
docs/observability.md	New documentation describing the telemetry architecture, signals, and deployment config.
docs/dev-guide.md	Updates Go prerequisite version.
docs/architecture.md	Updates repo structure + notes to include telemetry/OTel layers and context propagation updates.
AGENTS.md	Updates project structure and dependency documentation to include OTel components.
go.mod	Adds OTel dependencies and updates Go toolchain version directive.
go.sum	Updates dependency checksums corresponding to new/updated modules.

Comments suppressed due to low confidence (1)

pkg/core/statFileToCbDocMetParser.go:58

• If state.METParserNewDocId indicates a missing external doc reference, increment MissingExternalDocRefs here using the passed-in context so the metric can be correlated with the active span.

		telemetry.LinesParsed.Add(ctx, 1)
		state.METParserNewDocId = ""
		doc, err = parser.ParseLine(state.LoadSpec.DatasetName, headerLine, dataLine, &state.CbDocs, name, getMissingExternalDocForId)
		slog.Debug(fmt.Sprintf("OverWriteData:%v,METParserNewDocId:%v", state.LoadSpec.OverWriteData, state.METParserNewDocId))

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.